Category Archives: Systems

Upcoming changes during pentecost

+++ Update +++ www.cacert.org is now running on a new server, first tests were successful. Still some finetuning needs to be done afterwards +++ update +++

During the long weekend around pentecost (“Pfingsten” as it is called here in Germany) we’re planning the next step in replacing some hardware at the datacenter.

The main reason for the visit at the datacenter on monday is it to plug the serial connection between our webserver and signer to the new machine.

As our main website will move to a new server, which was installed in the datacenter during the last visit, there will be an interruption of service while doing the final copy and reconfiguration of the firewall (hopefully not longer than one hour).

While we’re at the datacenter we’re adding two SSD-drives to infra02. During the activation of the host system on these SSDs the services running on infra02 (like blog, wiki etc.) will not be accessible and/or slower than usual.

After all services are moved (remotely/afterwards) from the HDDs to SSDs everything should be active again … and most likely faster.

At a later visit (planned in July) the old sun1-server and old infra02-HDDs will be removed from the rack.

The final step for hardware-upgrade/replacement in the critical environment will be a replacement of the old signer machine(s) by new servers and HSM-modules. For this step software- as well as development team need some assistance in reviewing and testing especially the coding (written in Go). Feel free to contact us via support@.c.o, mailing-lists or using comments to this blog-entry.

Behind the scenes …

… we’ve just activated our own OCSP-resolver on our new arm64-servers.

This sounds a little bit unspectacular, but it’s a big milestone while replacing hard- and software within our environment as the old OCSP-resolver-software could not be ported to a recent debian and arm64-environment.

All other critical services (like Nameserver and CRL-Serving) were already moved successfully to our new power-saving machines (2 Raspberry Pi4) in the last weeks/months. OCSP needed some development and testing.

The virtual machines in the old environment are now stopped, within the next days the (power-consuming) sun3-server will then get it’s final shutdown and will be removed from CAcert-Rack during the next visit at the datacenter.

Our main website and signer-software will still be kept running on dedicated servers.

Upcoming Changes for www.cacert.org

Today we switched the connection to our main website as a preparation for a “bigger” change. Unfortunately this (temporary) change is not IPv6-capable, so only IPv4 is working currently.

Over the weekend we plan to move www.cacert.org to another server for a more recent environment and add a second firewall to our rack. During this server-transition you may face some issues while using www.cacert.org, after the weekend the services should be normal again.

Early next week we’ll enable IPv6 again for our main website (maybe by using a new IPv6-Address, but that’s not yet decided).

All other services (like blog/wiki/bugs/…) should remain active as usual as there is currently no planned update.

Nameserver-Changes for CAcert.org -update-

Update: Nameserver-transition is currently finished, new DNSSEC-records are set and active. KSK and ZSK were replaced by CSK.

In the ongoing process to update hard- and software we’re moving our main domain cacert.org to another master-nameserver-machine (with different nameserver-software) within our rack …

As we’re using DNSSEC to secure our domains, we need to update KSK and ZSK-keys for our domains during this progress, too.

Therefore you may face some DNSSEC-errors or issues in resolving cacert.org-domains within the next days, but this should resolve itself within some hours/days.

As soon as the transition of the nameserver-move is finished, I’ll update this post.

Todo: Give ns1.cacert.org the “old” nameserver-address again (after next hardware-change onsite) so secondary-nameserver ns3.cacert.org can get back to work. ns3 is currently not listed at our registrar, so not active for CAcert-Domains.

(Upcoming) work at the Datacenter

Update #1:

Moving www.cacert.org to new hardware was not successful due to some firewall settings, so we decided to keep the old server active.

During the next days/weeks we’ll change some firewall settings remotely so short downtimes may apply before we try to activate the new server during the next visit in some weeks.

Original note:

During the next visit at the datacenter on Friday we’re doing some hardware-changes within our rack, especially for our main website www.cacert.org.

As a preparation we will disable most of the services on www.cacert.org on Tuesday evening. The site will be fully operational again after the new server is up and running (most likely during Friday morning).

All other subdomains like blog/wiki/… will only have a short outage while we install a new firewall.

— this post will be updated after returning back from the datacenter —

When Captain CAcert rescues the Notaries of the Round Table

Today we are going on a little journey through time for a current occasion. Are you ready? Then jump into the fountain together with Frog King!

Many, many years ago, when grandmother was still a little girl, it may have been in 1995, a hardworking man named Mark Shuttleworth started a certificate issuing service in his poor parents’ garage, just like CAcert is one.

The name of this service was Thawte. Thawte was a great and important service. It is said that it covered half of the empire at that time. And because he was so old and so wise, he enjoyed some privileges. When Uncle Netscape, the browser, introduced new rules for certificates, Aunt Thawte, considering her age, only had to comply if she wanted to.

Now it was the case in those days that some people would have liked to send letters in an envelope. Good Aunt Thawte said: I have so many envelopes, I will give you some! And everyone who booked a free e-mail address with her got the certificate to wrap the messages as a gift. The Web of Trust was created to ensure that everything was above board and that the big bad wolf didn’t pretend to be one of the seven little goats. There, the letter writers met with the most trustworthy men and women of the entire empire for the knighting.

After the wizard Verisign took over Aunt Thawte’s service in 1999, the Web of Trust’s noble round table was abolished a few years later. Its members were very surprised to be thrown out of the castle just like that, since they had selflessly served the cause as noble knights and notaries.

However, it was a stormy time. And the storm wind blew a big sailing ship with full rigging from New South Wales, a spot of earth on a big island in the middle of the big, wide sea in the New World, across the ocean. Its name was emblazoned in gold letters on the stern: CAcert.

The captain held the wheel with both hands until the ship docked in a safe harbour. Immediately the crew rushed ashore to the desperate notaries and knights of the Thawte Round Table and offered to take them in their ship.

Numerous were those who gratefully accepted this offer, even more so when the captain said that he trusted Aunt Thawte. So it happened that large parts of Thawt’s Web of Trust were integrated into CAcert’s Web of Trust and the Thawte notaries became CAcert assurers. In a special program named Tverfiy, they could have their trust points transferred in 2009. Today, more than a decade later, CAcert is discontinuing the corresponding web site, after a long time since scattered notaries have joined CAcert’s community.

Further reading:
https://wiki.cacert.org/Tverify
https://en.wikipedia.org/wiki/Thawte#Web_of_Trust
old blog posts from the time

Signature server back in operation

Retour en fonctionnement du serveur de signature

Le serveur responsable de signer à la demande les certificats émis par CAcert dispose de deux disques durs, en redondance l’un de l’autre. Lorsqu’un dysfonctionnement se produit, aucune maintenance à distance n’est possible, car la machine n’est intentionnellement pas branchée au réseau. Seul un câble série permet d’échanger requêtes et réponses avec le reste de notre infrastructure. Aucune connexion n’est possible par ce moyen.

Or, depuis le 2 Août, nous observions la mise en attente de toutes les demandes de signature de certificats. L’équipe des infrastructures critiques est donc intervenue sur site ce 21 Août. Un problème dans le traitement d’un des certificats était la cause du blocage. Ce problème est résolu, mais reste à diagnostiquer avec précision. Il s’agit d’une série d’incidents que nous n’avions jamais vus auparavant.

Compte tenu des deux autres incidents intervenus plus tôt cette année, liés au système de fichiers de notre serveur de signature, nous devions accroitre sa résilience. Aussi, ce 21 août, l’équipe des infrastructures critiques a installé dans le rack un second serveur de signature, comme secours passif du premier. La présence de liens série dédiés vers chaque machine permettra à l’avenir de basculer très rapidement sur le second serveur de signature, en cas de nouveau problème. Dans tous les cas, les deux serveurs restent comme auparavant isolés du réseau.

Nous prions nos membres de nous excuser pour ces dysfonctionnements, et encourageons ceux résidant en Hollande où dans sa proche périphérie, à envisager de s’associer au travail de notre équipe des infrastructures critiques, ce qui augmenterait notre capacité d’intervention rapide.

Simultanément, nous espérons que l’intervention d’hier marque la fin de cette longue et exceptionnelle série.

English version

The server responsible for signing certificates issued by CAcert on demand has two hard disks, redundant to each other. When a malfunction occurs, no remote maintenance is possible, as the machine is intentionally not connected to the network. Only a serial cable is used to exchange requests and responses with the rest of our infrastructure. No connection is possible by this means.

However, since the 2nd of August, we have been seeing all certificate signing requests being put on hold. The Critical Infrastructure team therefore intervened on site on the 21st of August. A problem in the processing of one of the certificates was the cause of the blockage. This problem has been solved, but remains to be precisely diagnosed. This is a series of failures that we have never seen before.

In light of the two other incidents earlier this year related to the file system of our signature server, we needed to increase its resilience. So on 21 August, the Critical Infrastructure team installed a second signature server in the rack as a passive backup to the first. The presence of dedicated serial links to each machine will make it possible in future to switch very quickly to the second signature server in the event of a new problem. In any case, the two servers remain isolated from the network as before.

We apologise to our members for the inconvenience, and encourage those living in or near the Netherlands to consider working with our Critical Infrastructure team, which would increase our ability to respond quickly.

At the same time, we hope that yesterday’s intervention marks the end of this long and exceptional series.

New signer proves itself in use

EN: Signer is running again

DE: Signer ist wieder in Betrieb

FR: Signataire fonctionne à nouveau

ES: Firmante vuelve a funcionar

IT: Firmatario è di nuovo in funzione

The signer has been running again since yesterday, Friday, around 13:00 CEST. We then (while we were doing other work) watched the processing for about another hour… Around 0:30 CEST all outstanding certificate requests (~3000) were processed.

Things didn’t quite go as planned in June. As soon as something cannot be done remotely – there is no remote access to critical systems for security reasons – someone who is authorised to do so has to go the data centre in the Netherlands. Despite Corona, quarantine, floods, overtime at the company and whatever else comes up. That’s maybe two hours. Then two hours home again and in between the actual work. During the opening hours of the data centre, in your free time and paying for your own train ticket or petrol. It’s not always easy to reconcile all that. On Friday afternoon, however, the time had come and the Signer has now been running smoothly again for over a day.

As can be seen from the Critical Team’s plan published yesterday, preliminary work is already underway to make the system redundant throughout and even more robust, so that failures should no longer be noticed by users, because no one is interested in such failures! We are very sorry that you had to wait so long. At the same time, we thank the small core team who have sacrificed nights and weekends over the last five weeks to get the technology back up and running for the CAcert community!

Re-signed Class-3-Certificate – take action now!

English | Deutsch | Français | Español | Fingerprints

English | We already reported here in January that our Class 3 certificate is being re-signed. This was done a few weeks ago in our data centre in the Netherlands and subsequently tested extensively by our volunteers.

The new Class 3 certificate can now be downloaded here. In some days we will update the fingerprints and publish the other formats here. We recommend that all users use the new Class 3 certificate immediately, as the old certificate is approaching its expiry date and will no longer be valid after May 20th. Download the new certificate today and install it in your browser, e-mail program or certificate server as required.

All this exciting work (planning, re-signing, testing, communication) was done by volunteers from the CAcert community. They have acquired a lot of expertise over time and have worked their way up in the community. CAcert continues to offer such opportunities to interested and committed people today.

Alles neu macht der Mai – Neuerungen bei CAcert

Deutsch | Wir haben bereits im Januar an dieser Stelle darüber berichtet, dass unser Class-3-Zertifikat neu signiert wird. Dies ist vor einigen Wochen in unserem Rechenzentrum in den Niederlanden geschehen und anschliessend ausführlich von unseren Freiwilligen getestet worden.

Das neue Class-3-Zertifiat kann jetzt hier heruntergeladen werden. In wenigen Tagen werden wir die Fingerprints als auch die anderen Formate hier an gewohnter Stelle veröffentlichen. Wir empfehlen allen Nutzern, ab sofort das neue Class-3-Zertifikat zu verwenden, da das alte Zertifikate seinen Ablaufdatum entgegenschreitet und dann nicht mehr gültig ist. Laden Sie heute noch das neue Zertifikat herunter und installieren Sie es je nach Bedarf in Ihrem Browser, e-Mailprogramm oder Zertifikatsserver.

Alle diese spannenden Arbeiten (Planung, neu signieren, testen, Kommunikation) wurden von Freiwilligen der CAcert-Gemeinschaft erledigt. Sie haben sich im Laufe der Zeit viel Fachwissen angeeignet und sich in der Gemeinschaft hochgearbeitet. CAcert bietet auch heute interessierten und engagierten Leuten solche Möglichkeiten.

Changez vers le nouveau certificat class 3
Français | Nous avons déjà signalé ici en janvier que notre certificat de classe 3 était en cours de re-signature. Cela a été fait il y a quelques semaines dans notre centre de données aux Pays-Bas et a ensuite été testé de manière approfondie par nos volontaires.

Le nouveau certificat de classe 3 (comme l’ancien) peut être téléchargé ici. L’empreinte digitale va être publié dans les jours à venir ici. Nous recommandons à tous les utilisateurs d’utiliser le nouveau certificat de classe 3 à partir de maintenant, car l’ancien certificat approche de sa date d’expiration et ne sera plus valide. Téléchargez le nouveau certificat aujourd’hui et installez-le dans votre navigateur, votre programme de messagerie ou votre serveur de certificats, selon vos besoins.

Tout ce travail passionnant (planification, re-signature, tests, communication) a été réalisé par des bénévoles de la communauté CAcert. Ils ont acquis une grande expertise au fil du temps et ont gravi les échelons au sein de la communauté. CAcert continue aujourd’hui à offrir de telles opportunités aux personnes intéressées et engagées.

Español | Hace unas semanas, nuestro certificado de clase 3 se volvió a firmar en nuestro centro de datos de los Países Bajos y, a continuación, nuestros voluntarios lo probaron exhaustivamente. El nuevo certificado de clase 3 puede descargarse aquí. La huella dactilar y los demás formatos estarán disponibles en los próximos días aquí. Recomendamos a todos los usuarios que utilicen el nuevo certificado de clase 3 a partir de ahora, ya que el antiguo dejará de ser válido en breve. Instale el nuevo certificado de clase 3 hoy mismo.

Todo este apasionante trabajo ha sido realizado por voluntarios de la comunidad CAcert. CAcert ofrece interesantes oportunidades a las personas interesadas y dedicadas.

Fingerprints | SHA1 Fingerprint = D8:A8:3A:64:11:7F:FD:21:94:FE:E1:98:3D:D2:5C:7B:32:A8:FF:C8
SHA256 Fingerprint = 1B:C5:A6:1A:2C:0C:01:32:C5:2B:28:4F:3D:A0:D8:DA:CF:71:7A:0F:6C:1D:DF:81:D8:0B:36:EE:E4:44:28:69

Report of visit at datacenter on 2021-04-19

After a new member was added to the access engineers team it was possible to visit the datacenter following the epidemiological guidelines for SARS-CoV-2, as well as our own security guidelines.

During this visit we applied the long-awaited patch for bug 1438 by adding the serial number to certificate revocation lists.

This visit also provided an opportunity to instal a new infrastructure-server, courtesy of Abil’I.T. , a Luxembourg based free software service provider. Many thanks again!

… and …

We did the Class-3-resigning during this visit. Currently we’re testing this new Class-3-certificate and will publish it real soon.

A new visit in the summer will be necessary to replace hardware (and maybe apply further patches on the signer).