Category Archives: Systems

Certificate renewing is pending (update & help)

Some of our community members (users) get a problem while they try to renew an existing certificate. The issue is: Certificate renewal is pending for days/weeks.

First of all, CAcert is not a service provider or a company, but a community. We are all in the same boat. We can only achieve our goals together, with your the cooperation of all of us (of all users=members).

One of our volunteer support engineers, a retired gentleman somewhere in Bohemia, wrote, after he watererd the flowers in the garden:
1. Many users use CAcert without any assurance. Until now, their CSRs were signed by Class 1 Root (–> serial # 1xxxxx) and their CSRs/renewals are stuck in a queue now.
2. These users know absolutely nothing about existence Class 1 & Class 3 Roots, as they don’t remember installing root(s), and when creating a new cert, they cannot see the choice Class 1/3, because with <50 assurance points (trust points) it isn’t displayed.
3. Many users do not know about the existence of Wiki, bugs, blog, CATS… websites. Our education possibly fails in this direction.

And from Alsace, a baker who is also CAcert volunteer writes after putting the children to bed: There is a lot of information and many tutorials are at the FAQ at https://wiki.cacert.org How to create a certificate can be found at: https://wiki.cacert.org/HowTo/ClientCertCreate/

Another help message was sent by a CAcert volunteer who works as a bus driver from his mobile phone during the short break at the terminus: To get assurance points, the easyest way is to meet with two (or three) experienced assurers who can then credit you with the assurance (trust) points you need (you need 50 and get 10-35 per assurer). When you are on cacert.org in your account, go to the Web Of Trust: https://www.cacert.org/wot.php?id=12 (here you can enter your town and search for assurers in the area) or: https://www.cacert.org/wot.php?id=1 (here you can click through to choose from about 6000 assurers worldwide).

Thank you very much to all our active community members who helps here and there and gives other community members a hand. Even very little help is helpfull. If e.g. each of the 6000 assurers from the assurer directory helps with something small for 10 minutes per month, that is already 1000 hours of work. That would solve (almost) all problems. Here is how you too can give your CAcert community a hand: https://wiki.cacert.org/engagement

And another volunteer from Sweden points out, that the issue will not go away till the interface is fixed, which is a work that has been started, but not finished. Furthermore, renewing old incorrectly signed certificates will never work again, as we have said we will not fix the broken code for that, as no certificates should ever have been signed that way. We can’t continue signing them incorrectly.

Critical servers upgrade project

As a faulty connection cable between www.cacert.org and the signer made it necessary to travel to the datacenter this weekend instead of the planned schedule later this year we were able to finish this part earlier than expected: We finalized on the last steps of moving CAcert to a more modem hardware and software on critical servers.

This project was started “somehow” in May 2020 when the signer power board broke just before the Corona-Lockdown took place. The old signer was replaced by the same model at this visit. Since then we had several outages, which were mainly caused by broken hardware, sometimes noticed by our members, sometimes only visible in our internal monitoring.

Today the last of the old servers (our signer) was powered down as it was replaced by two modern machines using a more recent debian release, but keeping the old signer-coding.

The complete hardware-replacement-project reduced the power consumption of all CAcert-servers for more than 60%.

But that’s not all: We have plans to put our signer-environment to a new software written in Go, but here we need YOUR help in testing and reviewing the code. Feel free to contact support@cacert.org to get in touch to our experts.

Screenshot of the CAcert browser client certificate web application

Lowering the barriers of entry

In the coming few months we will start running some services with Let’s Encrypt server certificates. We decided to go this route to make it easier for people to join our community or contribute to our work.

A nice side effect of this move will be that we can provide these services https encrypted and redirect all unencrypted http URLs to their https counterparts.

We will continue to use our own server certificates for our CA systems and other services that are only relevant after joining our community.

We also will continue to provide our community with client and server certificates. All our services that support or require client certificates will still use those issued by our CA.

We recently implemented a web application to make it easier to get started with client certificates. The application provides a friendly and completely client side interface to generate key pairs and signing requests in your browser.

Upcoming changes during pentecost

+++ Update +++ www.cacert.org is now running on a new server, first tests were successful. Still some finetuning needs to be done afterwards +++ update +++

During the long weekend around pentecost (“Pfingsten” as it is called here in Germany) we’re planning the next step in replacing some hardware at the datacenter.

The main reason for the visit at the datacenter on monday is it to plug the serial connection between our webserver and signer to the new machine.

As our main website will move to a new server, which was installed in the datacenter during the last visit, there will be an interruption of service while doing the final copy and reconfiguration of the firewall (hopefully not longer than one hour).

While we’re at the datacenter we’re adding two SSD-drives to infra02. During the activation of the host system on these SSDs the services running on infra02 (like blog, wiki etc.) will not be accessible and/or slower than usual.

After all services are moved (remotely/afterwards) from the HDDs to SSDs everything should be active again … and most likely faster.

At a later visit (planned in July) the old sun1-server and old infra02-HDDs will be removed from the rack.

The final step for hardware-upgrade/replacement in the critical environment will be a replacement of the old signer machine(s) by new servers and HSM-modules. For this step software- as well as development team need some assistance in reviewing and testing especially the coding (written in Go). Feel free to contact us via support@.c.o, mailing-lists or using comments to this blog-entry.

Behind the scenes …

… we’ve just activated our own OCSP-resolver on our new arm64-servers.

This sounds a little bit unspectacular, but it’s a big milestone while replacing hard- and software within our environment as the old OCSP-resolver-software could not be ported to a recent debian and arm64-environment.

All other critical services (like Nameserver and CRL-Serving) were already moved successfully to our new power-saving machines (2 Raspberry Pi4) in the last weeks/months. OCSP needed some development and testing.

The virtual machines in the old environment are now stopped, within the next days the (power-consuming) sun3-server will then get it’s final shutdown and will be removed from CAcert-Rack during the next visit at the datacenter.

Our main website and signer-software will still be kept running on dedicated servers.

Upcoming Changes for www.cacert.org

Today we switched the connection to our main website as a preparation for a “bigger” change. Unfortunately this (temporary) change is not IPv6-capable, so only IPv4 is working currently.

Over the weekend we plan to move www.cacert.org to another server for a more recent environment and add a second firewall to our rack. During this server-transition you may face some issues while using www.cacert.org, after the weekend the services should be normal again.

Early next week we’ll enable IPv6 again for our main website (maybe by using a new IPv6-Address, but that’s not yet decided).

All other services (like blog/wiki/bugs/…) should remain active as usual as there is currently no planned update.

Nameserver-Changes for CAcert.org -update-

Update: Nameserver-transition is currently finished, new DNSSEC-records are set and active. KSK and ZSK were replaced by CSK.

In the ongoing process to update hard- and software we’re moving our main domain cacert.org to another master-nameserver-machine (with different nameserver-software) within our rack …

As we’re using DNSSEC to secure our domains, we need to update KSK and ZSK-keys for our domains during this progress, too.

Therefore you may face some DNSSEC-errors or issues in resolving cacert.org-domains within the next days, but this should resolve itself within some hours/days.

As soon as the transition of the nameserver-move is finished, I’ll update this post.

Todo: Give ns1.cacert.org the “old” nameserver-address again (after next hardware-change onsite) so secondary-nameserver ns3.cacert.org can get back to work. ns3 is currently not listed at our registrar, so not active for CAcert-Domains.

(Upcoming) work at the Datacenter

Update #1:

Moving www.cacert.org to new hardware was not successful due to some firewall settings, so we decided to keep the old server active.

During the next days/weeks we’ll change some firewall settings remotely so short downtimes may apply before we try to activate the new server during the next visit in some weeks.

Original note:

During the next visit at the datacenter on Friday we’re doing some hardware-changes within our rack, especially for our main website www.cacert.org.

As a preparation we will disable most of the services on www.cacert.org on Tuesday evening. The site will be fully operational again after the new server is up and running (most likely during Friday morning).

All other subdomains like blog/wiki/… will only have a short outage while we install a new firewall.

— this post will be updated after returning back from the datacenter —

When Captain CAcert rescues the Notaries of the Round Table

Today we are going on a little journey through time for a current occasion. Are you ready? Then jump into the fountain together with Frog King!

Many, many years ago, when grandmother was still a little girl, it may have been in 1995, a hardworking man named Mark Shuttleworth started a certificate issuing service in his poor parents’ garage, just like CAcert is one.

The name of this service was Thawte. Thawte was a great and important service. It is said that it covered half of the empire at that time. And because he was so old and so wise, he enjoyed some privileges. When Uncle Netscape, the browser, introduced new rules for certificates, Aunt Thawte, considering her age, only had to comply if she wanted to.

Now it was the case in those days that some people would have liked to send letters in an envelope. Good Aunt Thawte said: I have so many envelopes, I will give you some! And everyone who booked a free e-mail address with her got the certificate to wrap the messages as a gift. The Web of Trust was created to ensure that everything was above board and that the big bad wolf didn’t pretend to be one of the seven little goats. There, the letter writers met with the most trustworthy men and women of the entire empire for the knighting.

After the wizard Verisign took over Aunt Thawte’s service in 1999, the Web of Trust’s noble round table was abolished a few years later. Its members were very surprised to be thrown out of the castle just like that, since they had selflessly served the cause as noble knights and notaries.

However, it was a stormy time. And the storm wind blew a big sailing ship with full rigging from New South Wales, a spot of earth on a big island in the middle of the big, wide sea in the New World, across the ocean. Its name was emblazoned in gold letters on the stern: CAcert.

The captain held the wheel with both hands until the ship docked in a safe harbour. Immediately the crew rushed ashore to the desperate notaries and knights of the Thawte Round Table and offered to take them in their ship.

Numerous were those who gratefully accepted this offer, even more so when the captain said that he trusted Aunt Thawte. So it happened that large parts of Thawt’s Web of Trust were integrated into CAcert’s Web of Trust and the Thawte notaries became CAcert assurers. In a special program named Tverfiy, they could have their trust points transferred in 2009. Today, more than a decade later, CAcert is discontinuing the corresponding web site, after a long time since scattered notaries have joined CAcert’s community.

Further reading:
https://wiki.cacert.org/Tverify
https://en.wikipedia.org/wiki/Thawte#Web_of_Trust
old blog posts from the time

Signature server back in operation

Retour en fonctionnement du serveur de signature

Le serveur responsable de signer à la demande les certificats émis par CAcert dispose de deux disques durs, en redondance l’un de l’autre. Lorsqu’un dysfonctionnement se produit, aucune maintenance à distance n’est possible, car la machine n’est intentionnellement pas branchée au réseau. Seul un câble série permet d’échanger requêtes et réponses avec le reste de notre infrastructure. Aucune connexion n’est possible par ce moyen.

Or, depuis le 2 Août, nous observions la mise en attente de toutes les demandes de signature de certificats. L’équipe des infrastructures critiques est donc intervenue sur site ce 21 Août. Un problème dans le traitement d’un des certificats était la cause du blocage. Ce problème est résolu, mais reste à diagnostiquer avec précision. Il s’agit d’une série d’incidents que nous n’avions jamais vus auparavant.

Compte tenu des deux autres incidents intervenus plus tôt cette année, liés au système de fichiers de notre serveur de signature, nous devions accroitre sa résilience. Aussi, ce 21 août, l’équipe des infrastructures critiques a installé dans le rack un second serveur de signature, comme secours passif du premier. La présence de liens série dédiés vers chaque machine permettra à l’avenir de basculer très rapidement sur le second serveur de signature, en cas de nouveau problème. Dans tous les cas, les deux serveurs restent comme auparavant isolés du réseau.

Nous prions nos membres de nous excuser pour ces dysfonctionnements, et encourageons ceux résidant en Hollande où dans sa proche périphérie, à envisager de s’associer au travail de notre équipe des infrastructures critiques, ce qui augmenterait notre capacité d’intervention rapide.

Simultanément, nous espérons que l’intervention d’hier marque la fin de cette longue et exceptionnelle série.

English version

The server responsible for signing certificates issued by CAcert on demand has two hard disks, redundant to each other. When a malfunction occurs, no remote maintenance is possible, as the machine is intentionally not connected to the network. Only a serial cable is used to exchange requests and responses with the rest of our infrastructure. No connection is possible by this means.

However, since the 2nd of August, we have been seeing all certificate signing requests being put on hold. The Critical Infrastructure team therefore intervened on site on the 21st of August. A problem in the processing of one of the certificates was the cause of the blockage. This problem has been solved, but remains to be precisely diagnosed. This is a series of failures that we have never seen before.

In light of the two other incidents earlier this year related to the file system of our signature server, we needed to increase its resilience. So on 21 August, the Critical Infrastructure team installed a second signature server in the rack as a passive backup to the first. The presence of dedicated serial links to each machine will make it possible in future to switch very quickly to the second signature server in the event of a new problem. In any case, the two servers remain isolated from the network as before.

We apologise to our members for the inconvenience, and encourage those living in or near the Netherlands to consider working with our Critical Infrastructure team, which would increase our ability to respond quickly.

At the same time, we hope that yesterday’s intervention marks the end of this long and exceptional series.