Category Archives: Systems

Visit at the Datacenter on 2024-02-02

Today we visited the datacenter again to return the newly installed backup machine webdb2 and verify some settings on the signer machines.

While we were onsite, we updated neary all critical machines (including our main firewall), which caused outages of some minutes of our services.

After all updates were done we ran some tests including issuing Class1 and Class3-certifcates (client and server). A minor isse with crl-server (not running the rsync-service) was corrected afterwards remotely.

Both signers are now communicating with their webdb-servers. In the next days we’ll set up an automatic backup of webdb1 to webdb2 so webdb2 can fully replace webdb1 in case of a failure, which could not be corrected remoted.

Client certificate login temporarily not possible

1 Reply

Today we had been informed about an issue with client certificates issued on or after December 20, 2023 being used for login to www.cacert.org.

We did immediately switched off the login to www.CAcert.org for investigation.

We just activated the Username/Password-Login again, but keep the certificate login closed until the issue is resolved.

We will give an update with more details and plan to active certificate login again as soon as the issue is fixed.

CAcert Services mostly running again

Leave a reply

In Wednesday another visit at the datacenter took place, where we installed the updated webdb1-machine to the rack.

There are still some minor issues left (e.g. language selection for main website, automatic mails), which will be activated again remotely withins the next days.

This time the available time on critical teams site was blocked by some investigation issues (e.g.: What caused the outage, why did the internal routines and raid did not work) and non CAcert-related issues (as we all have a family and job, which are time-consuming as well) and and outage of usable internet-connection on critical teams site.

Naming this: If you’re living next to or in Netherlands and want to give us a helping hand for infrastructure and (possible) critical team feel free to contact us via support.

Certificate renewing is pending (update & help)

Leave a reply

Some of our community members (users) get a problem while they try to renew an existing certificate. The issue is: Certificate renewal is pending for days/weeks.

First of all, CAcert is not a service provider or a company, but a community. We are all in the same boat. We can only achieve our goals together, with your the cooperation of all of us (of all users=members).

One of our volunteer support engineers, a retired gentleman somewhere in Bohemia, wrote, after he watererd the flowers in the garden:
1. Many users use CAcert without any assurance. Until now, their CSRs were signed by Class 1 Root (–> serial # 1xxxxx) and their CSRs/renewals are stuck in a queue now.
2. These users know absolutely nothing about existence Class 1 & Class 3 Roots, as they don’t remember installing root(s), and when creating a new cert, they cannot see the choice Class 1/3, because with <50 assurance points (trust points) it isn’t displayed.
3. Many users do not know about the existence of Wiki, bugs, blog, CATS… websites. Our education possibly fails in this direction.

And from Alsace, a baker who is also CAcert volunteer writes after putting the children to bed: There is a lot of information and many tutorials are at the FAQ at https://wiki.cacert.org How to create a certificate can be found at: https://wiki.cacert.org/HowTo/ClientCertCreate/

Another help message was sent by a CAcert volunteer who works as a bus driver from his mobile phone during the short break at the terminus: To get assurance points, the easyest way is to meet with two (or three) experienced assurers who can then credit you with the assurance (trust) points you need (you need 50 and get 10-35 per assurer). When you are on cacert.org in your account, go to the Web Of Trust: https://www.cacert.org/wot.php?id=12 (here you can enter your town and search for assurers in the area) or: https://www.cacert.org/wot.php?id=1 (here you can click through to choose from about 6000 assurers worldwide).

Thank you very much to all our active community members who helps here and there and gives other community members a hand. Even very little help is helpfull. If e.g. each of the 6000 assurers from the assurer directory helps with something small for 10 minutes per month, that is already 1000 hours of work. That would solve (almost) all problems. Here is how you too can give your CAcert community a hand: https://wiki.cacert.org/engagement

And another volunteer from Sweden points out, that the issue will not go away till the interface is fixed, which is a work that has been started, but not finished. Furthermore, renewing old incorrectly signed certificates will never work again, as we have said we will not fix the broken code for that, as no certificates should ever have been signed that way. We can’t continue signing them incorrectly.

Critical servers upgrade project

Leave a reply

As a faulty connection cable between www.cacert.org and the signer made it necessary to travel to the datacenter this weekend instead of the planned schedule later this year we were able to finish this part earlier than expected: We finalized on the last steps of moving CAcert to a more modem hardware and software on critical servers.

This project was started “somehow” in May 2020 when the signer power board broke just before the Corona-Lockdown took place. The old signer was replaced by the same model at this visit. Since then we had several outages, which were mainly caused by broken hardware, sometimes noticed by our members, sometimes only visible in our internal monitoring.

Today the last of the old servers (our signer) was powered down as it was replaced by two modern machines using a more recent debian release, but keeping the old signer-coding.

The complete hardware-replacement-project reduced the power consumption of all CAcert-servers for more than 60%.

But that’s not all: We have plans to put our signer-environment to a new software written in Go, but here we need YOUR help in testing and reviewing the code. Feel free to contact support@cacert.org to get in touch to our experts.

Lowering the barriers of entry

Leave a reply

In the coming few months we will start running some services with Let’s Encrypt server certificates. We decided to go this route to make it easier for people to join our community or contribute to our work.

A nice side effect of this move will be that we can provide these services https encrypted and redirect all unencrypted http URLs to their https counterparts.

We will continue to use our own server certificates for our CA systems and other services that are only relevant after joining our community.

We also will continue to provide our community with client and server certificates. All our services that support or require client certificates will still use those issued by our CA.

We recently implemented a web application to make it easier to get started with client certificates. The application provides a friendly and completely client side interface to generate key pairs and signing requests in your browser.

Upcoming changes during pentecost

1 Reply

+++ Update +++ www.cacert.org is now running on a new server, first tests were successful. Still some finetuning needs to be done afterwards +++ update +++

During the long weekend around pentecost (“Pfingsten” as it is called here in Germany) we’re planning the next step in replacing some hardware at the datacenter.

The main reason for the visit at the datacenter on monday is it to plug the serial connection between our webserver and signer to the new machine.

As our main website will move to a new server, which was installed in the datacenter during the last visit, there will be an interruption of service while doing the final copy and reconfiguration of the firewall (hopefully not longer than one hour).

While we’re at the datacenter we’re adding two SSD-drives to infra02. During the activation of the host system on these SSDs the services running on infra02 (like blog, wiki etc.) will not be accessible and/or slower than usual.

After all services are moved (remotely/afterwards) from the HDDs to SSDs everything should be active again … and most likely faster.

At a later visit (planned in July) the old sun1-server and old infra02-HDDs will be removed from the rack.

The final step for hardware-upgrade/replacement in the critical environment will be a replacement of the old signer machine(s) by new servers and HSM-modules. For this step software- as well as development team need some assistance in reviewing and testing especially the coding (written in Go). Feel free to contact us via support@.c.o, mailing-lists or using comments to this blog-entry.

Behind the scenes …

Leave a reply

… we’ve just activated our own OCSP-resolver on our new arm64-servers.

This sounds a little bit unspectacular, but it’s a big milestone while replacing hard- and software within our environment as the old OCSP-resolver-software could not be ported to a recent debian and arm64-environment.

All other critical services (like Nameserver and CRL-Serving) were already moved successfully to our new power-saving machines (2 Raspberry Pi4) in the last weeks/months. OCSP needed some development and testing.

The virtual machines in the old environment are now stopped, within the next days the (power-consuming) sun3-server will then get it’s final shutdown and will be removed from CAcert-Rack during the next visit at the datacenter.

Our main website and signer-software will still be kept running on dedicated servers.

Upcoming Changes for www.cacert.org

2 Replies

Today we switched the connection to our main website as a preparation for a “bigger” change. Unfortunately this (temporary) change is not IPv6-capable, so only IPv4 is working currently.

Over the weekend we plan to move www.cacert.org to another server for a more recent environment and add a second firewall to our rack. During this server-transition you may face some issues while using www.cacert.org, after the weekend the services should be normal again.

Early next week we’ll enable IPv6 again for our main website (maybe by using a new IPv6-Address, but that’s not yet decided).

All other services (like blog/wiki/bugs/…) should remain active as usual as there is currently no planned update.

Nameserver-Changes for CAcert.org -update-

Leave a reply

Update: Nameserver-transition is currently finished, new DNSSEC-records are set and active. KSK and ZSK were replaced by CSK.

In the ongoing process to update hard- and software we’re moving our main domain cacert.org to another master-nameserver-machine (with different nameserver-software) within our rack …

As we’re using DNSSEC to secure our domains, we need to update KSK and ZSK-keys for our domains during this progress, too.

Therefore you may face some DNSSEC-errors or issues in resolving cacert.org-domains within the next days, but this should resolve itself within some hours/days.

As soon as the transition of the nameserver-move is finished, I’ll update this post.

Todo: Give ns1.cacert.org the “old” nameserver-address again (after next hardware-change onsite) so secondary-nameserver ns3.cacert.org can get back to work. ns3 is currently not listed at our registrar, so not active for CAcert-Domains.