Category Archives: Progress

Upcoming changes during pentecost

+++ Update +++ www.cacert.org is now running on a new server, first tests were successful. Still some finetuning needs to be done afterwards +++ update +++

During the long weekend around pentecost (“Pfingsten” as it is called here in Germany) we’re planning the next step in replacing some hardware at the datacenter.

The main reason for the visit at the datacenter on monday is it to plug the serial connection between our webserver and signer to the new machine.

As our main website will move to a new server, which was installed in the datacenter during the last visit, there will be an interruption of service while doing the final copy and reconfiguration of the firewall (hopefully not longer than one hour).

While we’re at the datacenter we’re adding two SSD-drives to infra02. During the activation of the host system on these SSDs the services running on infra02 (like blog, wiki etc.) will not be accessible and/or slower than usual.

After all services are moved (remotely/afterwards) from the HDDs to SSDs everything should be active again … and most likely faster.

At a later visit (planned in July) the old sun1-server and old infra02-HDDs will be removed from the rack.

The final step for hardware-upgrade/replacement in the critical environment will be a replacement of the old signer machine(s) by new servers and HSM-modules. For this step software- as well as development team need some assistance in reviewing and testing especially the coding (written in Go). Feel free to contact us via support@.c.o, mailing-lists or using comments to this blog-entry.

Upcoming Changes for www.cacert.org

Today we switched the connection to our main website as a preparation for a “bigger” change. Unfortunately this (temporary) change is not IPv6-capable, so only IPv4 is working currently.

Over the weekend we plan to move www.cacert.org to another server for a more recent environment and add a second firewall to our rack. During this server-transition you may face some issues while using www.cacert.org, after the weekend the services should be normal again.

Early next week we’ll enable IPv6 again for our main website (maybe by using a new IPv6-Address, but that’s not yet decided).

All other services (like blog/wiki/bugs/…) should remain active as usual as there is currently no planned update.

Nameserver-Changes for CAcert.org -update-

Update: Nameserver-transition is currently finished, new DNSSEC-records are set and active. KSK and ZSK were replaced by CSK.

In the ongoing process to update hard- and software we’re moving our main domain cacert.org to another master-nameserver-machine (with different nameserver-software) within our rack …

As we’re using DNSSEC to secure our domains, we need to update KSK and ZSK-keys for our domains during this progress, too.

Therefore you may face some DNSSEC-errors or issues in resolving cacert.org-domains within the next days, but this should resolve itself within some hours/days.

As soon as the transition of the nameserver-move is finished, I’ll update this post.

Todo: Give ns1.cacert.org the “old” nameserver-address again (after next hardware-change onsite) so secondary-nameserver ns3.cacert.org can get back to work. ns3 is currently not listed at our registrar, so not active for CAcert-Domains.

Signature server back in operation

Retour en fonctionnement du serveur de signature

Le serveur responsable de signer à la demande les certificats émis par CAcert dispose de deux disques durs, en redondance l’un de l’autre. Lorsqu’un dysfonctionnement se produit, aucune maintenance à distance n’est possible, car la machine n’est intentionnellement pas branchée au réseau. Seul un câble série permet d’échanger requêtes et réponses avec le reste de notre infrastructure. Aucune connexion n’est possible par ce moyen.

Or, depuis le 2 Août, nous observions la mise en attente de toutes les demandes de signature de certificats. L’équipe des infrastructures critiques est donc intervenue sur site ce 21 Août. Un problème dans le traitement d’un des certificats était la cause du blocage. Ce problème est résolu, mais reste à diagnostiquer avec précision. Il s’agit d’une série d’incidents que nous n’avions jamais vus auparavant.

Compte tenu des deux autres incidents intervenus plus tôt cette année, liés au système de fichiers de notre serveur de signature, nous devions accroitre sa résilience. Aussi, ce 21 août, l’équipe des infrastructures critiques a installé dans le rack un second serveur de signature, comme secours passif du premier. La présence de liens série dédiés vers chaque machine permettra à l’avenir de basculer très rapidement sur le second serveur de signature, en cas de nouveau problème. Dans tous les cas, les deux serveurs restent comme auparavant isolés du réseau.

Nous prions nos membres de nous excuser pour ces dysfonctionnements, et encourageons ceux résidant en Hollande où dans sa proche périphérie, à envisager de s’associer au travail de notre équipe des infrastructures critiques, ce qui augmenterait notre capacité d’intervention rapide.

Simultanément, nous espérons que l’intervention d’hier marque la fin de cette longue et exceptionnelle série.

English version

The server responsible for signing certificates issued by CAcert on demand has two hard disks, redundant to each other. When a malfunction occurs, no remote maintenance is possible, as the machine is intentionally not connected to the network. Only a serial cable is used to exchange requests and responses with the rest of our infrastructure. No connection is possible by this means.

However, since the 2nd of August, we have been seeing all certificate signing requests being put on hold. The Critical Infrastructure team therefore intervened on site on the 21st of August. A problem in the processing of one of the certificates was the cause of the blockage. This problem has been solved, but remains to be precisely diagnosed. This is a series of failures that we have never seen before.

In light of the two other incidents earlier this year related to the file system of our signature server, we needed to increase its resilience. So on 21 August, the Critical Infrastructure team installed a second signature server in the rack as a passive backup to the first. The presence of dedicated serial links to each machine will make it possible in future to switch very quickly to the second signature server in the event of a new problem. In any case, the two servers remain isolated from the network as before.

We apologise to our members for the inconvenience, and encourage those living in or near the Netherlands to consider working with our Critical Infrastructure team, which would increase our ability to respond quickly.

At the same time, we hope that yesterday’s intervention marks the end of this long and exceptional series.

New signer proves itself in use

EN: Signer is running again

DE: Signer ist wieder in Betrieb

FR: Signataire fonctionne à nouveau

ES: Firmante vuelve a funcionar

IT: Firmatario è di nuovo in funzione

The signer has been running again since yesterday, Friday, around 13:00 CEST. We then (while we were doing other work) watched the processing for about another hour… Around 0:30 CEST all outstanding certificate requests (~3000) were processed.

Things didn’t quite go as planned in June. As soon as something cannot be done remotely – there is no remote access to critical systems for security reasons – someone who is authorised to do so has to go the data centre in the Netherlands. Despite Corona, quarantine, floods, overtime at the company and whatever else comes up. That’s maybe two hours. Then two hours home again and in between the actual work. During the opening hours of the data centre, in your free time and paying for your own train ticket or petrol. It’s not always easy to reconcile all that. On Friday afternoon, however, the time had come and the Signer has now been running smoothly again for over a day.

As can be seen from the Critical Team’s plan published yesterday, preliminary work is already underway to make the system redundant throughout and even more robust, so that failures should no longer be noticed by users, because no one is interested in such failures! We are very sorry that you had to wait so long. At the same time, we thank the small core team who have sacrificed nights and weekends over the last five weeks to get the technology back up and running for the CAcert community!

Re-signed Class-3-Certificate – take action now!

English | Deutsch | Français | Español | Fingerprints

English | We already reported here in January that our Class 3 certificate is being re-signed. This was done a few weeks ago in our data centre in the Netherlands and subsequently tested extensively by our volunteers.

The new Class 3 certificate can now be downloaded here. In some days we will update the fingerprints and publish the other formats here. We recommend that all users use the new Class 3 certificate immediately, as the old certificate is approaching its expiry date and will no longer be valid after May 20th. Download the new certificate today and install it in your browser, e-mail program or certificate server as required.

All this exciting work (planning, re-signing, testing, communication) was done by volunteers from the CAcert community. They have acquired a lot of expertise over time and have worked their way up in the community. CAcert continues to offer such opportunities to interested and committed people today.

Alles neu macht der Mai – Neuerungen bei CAcert

Deutsch | Wir haben bereits im Januar an dieser Stelle darüber berichtet, dass unser Class-3-Zertifikat neu signiert wird. Dies ist vor einigen Wochen in unserem Rechenzentrum in den Niederlanden geschehen und anschliessend ausführlich von unseren Freiwilligen getestet worden.

Das neue Class-3-Zertifiat kann jetzt hier heruntergeladen werden. In wenigen Tagen werden wir die Fingerprints als auch die anderen Formate hier an gewohnter Stelle veröffentlichen. Wir empfehlen allen Nutzern, ab sofort das neue Class-3-Zertifikat zu verwenden, da das alte Zertifikate seinen Ablaufdatum entgegenschreitet und dann nicht mehr gültig ist. Laden Sie heute noch das neue Zertifikat herunter und installieren Sie es je nach Bedarf in Ihrem Browser, e-Mailprogramm oder Zertifikatsserver.

Alle diese spannenden Arbeiten (Planung, neu signieren, testen, Kommunikation) wurden von Freiwilligen der CAcert-Gemeinschaft erledigt. Sie haben sich im Laufe der Zeit viel Fachwissen angeeignet und sich in der Gemeinschaft hochgearbeitet. CAcert bietet auch heute interessierten und engagierten Leuten solche Möglichkeiten.

Changez vers le nouveau certificat class 3
Français | Nous avons déjà signalé ici en janvier que notre certificat de classe 3 était en cours de re-signature. Cela a été fait il y a quelques semaines dans notre centre de données aux Pays-Bas et a ensuite été testé de manière approfondie par nos volontaires.

Le nouveau certificat de classe 3 (comme l’ancien) peut être téléchargé ici. L’empreinte digitale va être publié dans les jours à venir ici. Nous recommandons à tous les utilisateurs d’utiliser le nouveau certificat de classe 3 à partir de maintenant, car l’ancien certificat approche de sa date d’expiration et ne sera plus valide. Téléchargez le nouveau certificat aujourd’hui et installez-le dans votre navigateur, votre programme de messagerie ou votre serveur de certificats, selon vos besoins.

Tout ce travail passionnant (planification, re-signature, tests, communication) a été réalisé par des bénévoles de la communauté CAcert. Ils ont acquis une grande expertise au fil du temps et ont gravi les échelons au sein de la communauté. CAcert continue aujourd’hui à offrir de telles opportunités aux personnes intéressées et engagées.

Español | Hace unas semanas, nuestro certificado de clase 3 se volvió a firmar en nuestro centro de datos de los Países Bajos y, a continuación, nuestros voluntarios lo probaron exhaustivamente. El nuevo certificado de clase 3 puede descargarse aquí. La huella dactilar y los demás formatos estarán disponibles en los próximos días aquí. Recomendamos a todos los usuarios que utilicen el nuevo certificado de clase 3 a partir de ahora, ya que el antiguo dejará de ser válido en breve. Instale el nuevo certificado de clase 3 hoy mismo.

Todo este apasionante trabajo ha sido realizado por voluntarios de la comunidad CAcert. CAcert ofrece interesantes oportunidades a las personas interesadas y dedicadas.

Fingerprints | SHA1 Fingerprint = D8:A8:3A:64:11:7F:FD:21:94:FE:E1:98:3D:D2:5C:7B:32:A8:FF:C8
SHA256 Fingerprint = 1B:C5:A6:1A:2C:0C:01:32:C5:2B:28:4F:3D:A0:D8:DA:CF:71:7A:0F:6C:1D:DF:81:D8:0B:36:EE:E4:44:28:69

Engineers nominated

The free certificate authority CAcert is making progress in increasing the number of its working groups. In the past few days, the committee approved the appointment of Jan to the post of Critical Engineer.

The appointment of Michaela as Access Engineer was also approved. Both have a broad range of experience and are distinguished by their specialist knowledge and sense of responsibility. We wish both engineers much success and fulfilment in their voluntary work for the CAcert community. These are challenging tasks and come with great responsibility. CAcert offers interested volunteers a variety of tasks, the opportunity to gain exciting experience and stimulating career opportunities.

New Background check etablished

English – Français – Deutsch

English Our efforts, most recently under Ted’s leadership, to re-establish a functioning background assessment system (formerly called ABC) for incoming volunteers were successful. This was essential for us to be able to staff our teams again.

In May-June 2020, after recognising the inability of the last remaining arbitrators to re-establish the arbitration function anywhere near what it was until 2016, and after consultation with the arbitrators, we established a new background check process, which we entrusted to members of the association who had undergone this ABC check themselves in the past. The first entrusted members to interview newcomers were Bernhard F and Dirk A.

As a result of this work, new background checks were carried out for the first time in more than seven years in the past weeks. Three candidates for positions with wide ranging system access were interviewed by the two examiners in a suitable interview setting to assess with them whether they might be open to external manipulation or even blackmail once appointed to positions with wide ranging access to the machines. Each of the applicants presented a report of their interview to the committee and the committee members were asked to vote on whether they should accept appointment to the positions of responsibility under consideration.

Français Nos efforts pour rétablir un système fonctionnel d’évaluation des antécédents (anciennement appelé ABC) pour les nouveaux volontaires ont été couronnés de succès. C’était indispensable pour que nous puissions reconstituer nos équipes. Après avoir reconnu l’incapacité des derniers arbitres restants à rétablir la fonction d’arbitrage à un niveau proche de ce qu’elle était jusqu’en 2016, nous avons établi un nouveau processus de vérification des antécédents (ABC) que nous avons confié à des membres de l’Association qui s’étaient eux-mêmes soumis à cet examen ABC dans le passé.

Grâce à ce travail, des vérifications des antécédents ont été effectuées pour la première fois en plus de sept ans au cours des dernières semaines. Trois candidats ont été interrogés dans un cadre d’entretien approprié afin d’évaluer avec eux s’ils pouvaient être ouverts à la manipulation externe ou même au chantage une fois nommés à des postes ayant un accès étendu aux machines. Les membres du comité ont été invités à voter pour accepter ou non la nomination aux postes de responsabilité envisagés après avoir vu les résultats.

Deutsch Unsere Bemühungen wieder eine funktionierende Hintergrundprüfung (früher ABC genannt) für neu hinzukommende Freiwillige einzuführen, waren erfolgreich. Dies war unerlässlich, damit wir unsere Arbeitsgruppen wieder besetzen können.

Nachdem wir die Unmöglichkeit erkannt hatten, die Schiedsfunktion wieder annähernd so zu etablieren, wie sie bis 2016 war, haben wir nach Rücksprache mit den Arbitratoren eine neue Hintergrundprüfung eingerichtet, mit der wir Mitglieder des Vereins betraut haben, die sich in der Vergangenheit bereits selbst dieser ABC-Überprüfung unterzogen hatten.

Deshalb wurden in den vergangenen Wochen nach mehr als sieben Jahren erstmals wieder Hintergrundprüfungen durchgeführt. Drei Kandidaten für Posten mit weitreichendem Systemzugang wurden von den beiden Prüfern in einem geeigneten Gesprächsrahmen befragt, um mit ihnen abzuschätzen, ob sie nach ihrer Ernennung in Positionen mit weitreichendem Zugang zu den Maschinen für externe Manipulationen oder gar Erpressungen offen sein könnten. Der Vorstand wurde daraufhin gebeten, nach Lektüre des Prüfberichtes, darüber zu beraten, ob er der Ernennung des Bewerbers für die in Frage kommende Verantwortungsposition annehmen kann.

OCSP is working well with Mozilla Firefox again

Rapperswil

There was a issue with ocsp.cacert.org on Mozilla Firefox browsers. We published a work around last year. In between a volunteer detected, that a script on ocsp.cacert.org was not running as expected. Therefore the CRL expired for OCSP-Daemon, hence giving the OLD_RESPONSE-error. We restarted the script allready in 2020 and changed our internal monitoring.

The OCSP responder has been restored to a proper working state and is monitored properly now. As it worked now for more than 3 months properly, the work around is no more needed.

New Class 3 certificate expected for May

Question of a member of the community: The Class3 certificate is set to expire this year in May. What do I need to do with:
a) My existing certificates that were signed against the Class 3
b) Installation of the new Class3 and when will that happen?

Answer from our volunteer critical system administrator: Thank you for this question. We started the resigning procedure already last year. We plan to use the same private key for the new class-3-certificate, so the old certificates will remain valid.

As the resigning needs to be done in the data centre in the Netherlands, this is planned for February/March – depending on the pandemic situation. So there will be enough time to replace the Class-3-Root in your configuration (Certificate-Chain) or your Browsers.

Deutsch: Die Planung, um das Class-3-Zertifikat neu zu signieren wurden bereits im letzten Jahr aufgenommen. Der Zeitplan ist grosszügig, dass die Arbeiten unserer Freiwilligen vor Mai abgeschlossen sein werden.

Français: Le projet de re-signer le certificat de la classe 3 a été lancé l’année dernière. Le calendrier est généreux, le travail de nos volontaires sera terminé avant le mois de mai.

Español: La planificación para volver a firmar el certificado de la clase 3 comenzó el año pasado. El programa es generoso, el trabajo de nuestros voluntarios se completará antes de mayo.

Português: O planejamento para assinar novamente o certificado de Classe 3 começou no ano passado. O cronograma é generoso, pois o trabalho de nossos voluntários será concluído antes de maio.