Tag Archives: Infrastructure

CAcert’s infrastructure ready for the future

On saturday, 13th of July 2019, in a joint operation, CAcert Infrastructure Team and CAcert Critical Team updated the operation system of CAcert’s infrastructure in the Netherlands sucessfully. The system is now running on the Debian Buster OS release that has been released by the Debian project last weekend.

Timing

The teams started this morning at around 9:30 CEST and finished the upgrades at 16:30 CEST, some of our applications turned back to service afterwards. The system is running smoothly now.

What is new?

The new OS release provides some features that are important for our infrastructure and will allow better operation of our applications in the future:

  • LXC has been upgraded from the somewhat primitive 0.8.0 pre-release to LXC 3.0.3 that has a proper API, better security and which will help application administrators
  • Firewalling/forwarding/NAT should now be faster then the old iptables setup. We still use ferm as a wrapper but the CAcert Infrastructure Team is already considering switching to native nftables rules that will provide a similar but faster rule set.
  • Further details about this major update can be read on our mailing list.

CAcert Infrastructure Team Lead JanDD is happy that we could finish this big upgrade and that we could implement all these changes for you. In a statement made on the early saturday evening, he thanked again to Wytze from CAcert Critical Team for his great support during the day.

The volunteers from these two teams worked for seven and a half hours today, Saturday, to keep our systems up to date. Join us in thanking them and donate now at your own discretion. Your donation will only be used to pay for the infrastructure (hosting, electricity in the data center). «I say thank you to Jan and Wytze and their team with a donation!»

If you find any issues that might be caused by the upgrade feel free to file bugs on https://bugs.cacert.org/ (at project Infrastructure > Infrastructue hosts).

If you want to join one of our teams, please join the development mailing list or write to the secretary.

Service interruption for a major system upgrade for a brighter future of CAcert on Sat, 13-07-2019

The CAcert Infrastructure Team will perform a major system upgrade of our infrastructure host tomorrow, 13th of July 2019, starting at 8am UTC/10am CEST. Wytze van der Raay of the critical infrastructure team will assist via remote console if necessary.

We expect the upgrade to run for at least 4 hours and some services might need fixes that will require even longer.

Most services will be unavailable at least for parts of the upgrade session. We will try to keep the downtime of essential services (email, emailout, lists, blog, wiki) as short as possible. We hope to not cause to many inconvenience but we cannot wait longer to perform these long needed update. The Debian Buster stable release last week and the recently acquired knowledge on how to use the remote console system of infra02 inspired us to perform the upgrade now.

Jan Dittberner
CAcert Infrastructure Team Lead

Please support the hudge work of the volunteers of our Infrastrucutre Team, please donate to continue to run this service. Thank you.

Stability of e-mail verification strongly improved

The e-mail verification on the CAcert web server has recently led to repeated support requests. An analysis of the log files in our data center showed that the corresponding error occurred more frequently. So we have to conclude that many CAcert users have been negatively affected. In order to avoid further negative effects, an emergency
patch was deemed necessary by the Critical System Administrator Team.

The standardised review and testing of the emergency patch implemented yesterday is carried out by the regular teams in the aftermath, which can result in a formal blessing for this patch or a request for additional code or configuration changes. We would like to thank the Critical System Administrator Team for their quick and decisive action. All teams consist of volunteers. If you want to support the work done by the Critical System Administration Team and the review by the Software Team, please donate, to continue to run this service. Thank you.

Updated: Information about Heartbleed-bug in OpenSSL 1.0.1 up to 1.0.1f

German version below

There is news about a bug in OpenSSL that may allow an attacker to leak arbitrary information from any process using OpenSSL.

Good news:

Certificates issued by CAcert are not broken and our central systems did not leak your keys.

Bad news:

Even then you may be affected.
Although your keys were not leaked by CAcert your keys on your own infrastructure systems might have been compromised if you were or are running a vulnerable version of OpenSSL.

To elaborate on this:

The central systems of CAcert and our root certificates are not affected by this issue. Regrettably some of our infrastructure systems were affected by the bug. We are working to fix them and already completed work for the most critical ones. If you logged into those systems, within the last two years, (see list below) you might be affected!
But unfortunately given the nature of this bug we have to assume that the certificates of our members may be affected, if they were used in an environment with a publically accessable OpenSSL connection (e.g. Apache webserver, mail server, Jabber-Server, …). The bug has been open in OpenSSL for two years – from December 2011 and was introduced in stable releases starting with OpenSSL 1.0.1.
When an attacker can reach a vulnerable service he can abuse the TLS heartbeat extension to retrieve arbitrary chunks of memory by exploiting a missing bounds check. This can lead to disclosure of your private keys, resident session keys and other key material as well as all volatile memory contents of the server process like passwords, transmitted user data (e.g. web content) as well as other potentially confidential information.
Exploiting this bug does not leave any noticeable traces, thus for any system which is (or has been) running a vulnerable version of OpenSSL you must assume that at least your used server keys are compromised and therefore must be replaced by newly generated ones. Simply renewing existing certificates is not sufficient! – Please generate NEW keys with at least 2048 bit RSA or stronger!
As mentioned above this bug can be used to leak passwords and thus you should consider changing your login credentials to potentially compromised systems as well as any other system where those credentials might have been used as soon as possible.
An (incomplete) list of commonly used software which include or link to OpenSSL can be found at related apps.

What to do?

  • First ensure that you upgrade your system to a fixed OpenSSL version (1.0.1g or above).
  • Only then create new keys for your certificates.
  • Check what services you have used that may have been affected within the last two years.
  • Wait until you think that those environments got fixed.
  • Then (and only then) change your credentials for those services. If you do it too early, i.e. before the sites got fixed, your data may be leaked, again. So be careful when you do this.

What we are doing:

  • We are updating the affected infrastructure systems and and create new certificates for them.
  • We use this opportunity to upgrade to 4096 bit RSA keys signed with SHA-512. The new fingerprints can be found below. 😉
  • We will contact all members, who had active server certificates within the last two years.
  • We will keep you updated, here.

Press release

CAcert-PM-Heartbleed-en

Update:

There is a website where one can check if a domain is affected:
http://filippo.io/Heartbleed/ (Use on your own risk)
We checked our systems against it, it looks like most of the systems we classified as affected are not actually affected. But we decided to update them, anyway and provide them with new certificates, as well.

Status of CAcert Infrastructure systems:

Not affected / Nicht betroffen:

  • main website (www.cacert.org)
  • certificate signer (the system doing the actual certificate issuing)
  • email system (inbound and outbound mail servers)

Fixed/Behoben:

blog.cacert.org
SHA1 Fingerprint=7A:13:19:98:1E:FB:9F:F9:9A:E6:3D:E5:7D:F0:42:E1:BE:56:B9:79

board.cacert.org
SHA1 Fingerprint=9B:30:44:3D:A8:8C:F5:5E:50:07:68:70:D6:A1:83:44:F9:7D:00:D6

bugs.cacert.org
SHA1 Fingerprint=E1:2C:12:7F:66:DF:2E:9D:F6:BC:FB:6F:BC:F1:2E:A0:10:5F:8E:BA

cats.cacert.org
SHA1 Fingerprint=2E:0B:08:6E:F1:48:FB:76:69:D6:6D:51:8F:E9:B3:2C:C3:4B:14:25

community.cacert.org
SHA1 Fingerprint=9B:8E:0A:68:96:E5:C8:E6:E6:8E:D8:10:31:3F:7C:2C:A8:4E:E1:3F

email.cacert.org
SHA1 Fingerprint=27:AB:9B:90:51:9B:BA:60:51:B3:84:54:FF:C7:09:94:63:86:68:FA

git.cacert.org
SHA1 Fingerprint=87:47:41:6D:C1:32:C7:22:00:E5:DA:E5:3C:4B:28:A2:2B:8A:F3:E4

irc.cacert.org
SHA1 Fingerprint=57:F4:0F:38:1E:53:D0:83:DC:D2:40:0A:13:98:B7:06:55:EA:A7:19

issue.cacert.org
SHA1 Fingerprint=82:33:D3:AE:32:56:C5:AD:9E:BF:D1:84:62:56:EA:95:31:7E:64:8C

lists.cacert.org
SHA1 Fingerprint=6A:AE:16:90:A2:1F:CC:1B:B7:93:71:C0:1B:BD:2E:14:68:69:45:EA

svn.cacert.org
SHA1 Fingerprint=D5:00:0A:15:17:04:2F:50:5B:09:3C:DD:B9:0A:57:DD:B3:BE:3D:B4

translations.cacert.org
SHA1 Fingerprint=B7:59:B9:BA:46:64:E2:D4:C8:73:20:50:45:9B:08:5E:2B:DF:D0:1B

wiki.cacert.org
SHA1 Fingerprint=87:07:59:30:30:64:27:15:6E:39:C3:66:09:CA:7A:90:7D:2F:32:32

cacert.eu
SHA1 Fingerprint=A2:7A:CB:E7:91:0A:ED:7E:63:9F:D1:97:01:96:E9:7B:F0:9E:43:3D

Affected/Betroffen:

  • Testserver-management-system (you should have not used correct data there)

Deutsche Fassung:
Ein Bug in OpenSSL wurde gefunden, der es einem Angreifer erlaubt beliebige Informationen jedes Prozesses zu erlangen, der OpenSSL nutzt.

Die gute Nachricht:

Die von CAcert ausgestellten Zertifikate sind nicht kaputt und unsere zentralen Systeme waren auch nicht angreifbar und hat keine Schlüssel verraten.

Die schlechte Nachricht:

Dennoch kann jeder betroffen sein!

Um ins Detail zu gehen:

Die zentralen Systeme und die Stammzertifikate von CAcert sind von diesem Problem nicht betroffen. Leider sind einige unserer Infrastruktur-Systeme durch den Fehler betroffen.
Wir arbeiten daran diese zu fixen und haben dies auch schon für die meisten erledigt. Jeder, der sich auf diese Systeme in den letzten zwei Jahre eingelogt hat kann betroffen sein!
Aufgrund der Art des Fehlers, müssen wir leider davon ausgehen, dass die Zertifikate unserer Mitglieder betroffen sind, wenn sie sich in eine Umgebung eingelogt haben, die über öffentliche OpenSSL-Verbindungen zugänglich war (z.B. Apache Webserver, Mail Server, Jaber-Server, …). Dieser Fehler war zwei Jahre lang in OpenSSL – seit Dezember 2011 – und kam beginnend mit Version 1.0.1 in die Stabilen Versionen.
Angreifer, die einen verwundbaren Service erreichen, können die TLS-Erweiterung “heartbeat” ausnutzen, um beliebige Abschnitte aus dem Speicher zu erlangen, indem sie eine fehlende Bereichsprüfung ausnutzen. Das kann zur Offenlegung von privaten Schlüsseln, im Speicher abgelegte Sitzungsschlüsseln, sonstige Schlüssel genauso wie jeglicher weiterer Speicherinhalt des Server-Prozesses wie Passwörter oder übermittelte Benutzerdaten (z.B. Webinhalte) oder andere vertrauliche Informationen führen.
Die Ausnutzung dieses Fehlers hinterlässt keine merklichen Spuren. Daher muss für jedes System, auf dem eine angreifbare Version von OpenSSL läuft (oder lief), angenommen werden, dass zumindest die verwendeten Server-Zertifikate kompromittiert sind und deswegen durch einen NEU generierte erstetzt werden müssen. Einfach die alten Zertifikate zu erneuern, reicht nicht aus! – Bitte NEUE Schlüssel mit 2048 Bit RSA oder stärker generieren!
Wie oben erwähnt kann dieser Fehler ausgenutzt werden, um Passwörter zu entwenden. Daher sollte jeder überlegen, alle Zugangsdaten zu möglicherweise betroffenen Systemen und allen Systemen bei denen diese sonst noch verwendet worden sein können, so bald wie möglich auszutauschen.
Eine (unvollständige) Liste an weit verbreiteter Software die OpenSSL verwendet kann z.B. unter folgendem Link gefunden werden.

Was ist zu tun?

  • Als erstes müssen die eigenen Systeme auf eine fehlerbereinigte Version von OpenSSL aktualisiert werden (Version 1.0.1g oder neuer).
  • Danach neue Schlüssel für die Zertifikate erstellen. Jetzt ist es sicher das zu tun.
  • Überprüfen, welche fremden Dienste in den letzten zwei Jahren besucht worden sind.
  • Warten, bis dort wahrscheinlich der Fehler behoben wurde.
  • Dann (und erst dann) die Logindaten für diese Dienste erneuern. Vorsicht: Wenn das zu früh getan wird, also wenn der Dienst noch nicht bereinigt wurde, können die Daten wieder abgegriffen werden.

Pressemitteilung

CAcert-PM-Heartbleed-de

Update:

Es gibt eine Webseite, bei der man seit kurzem checken kann, ob eine Domäne angreifbar sind:
http://filippo.io/Heartbleed/ (Benutzung auf eigene Gefahr)
Wir haben unsere Systeme dagegen geprüft und es sieht so aus, als wären nicht alle, die wir als potentiell angreifbar eingestuft haben tatsächlich angreifbar, aber wir haben uns dennoch dafür entschieden die entsprechenden Systeme aufzurüsten und die Zertifikate zu erneuern.

Was wir tun:

  • Wir arbeiten daran, alle Infrastruktur-Systeme auf den neuesten OpenSSL-Stand zu bringen und für diese neue Zertifikate zu generieren.
  • Wir nutzen diese Gelegenheit, um dabei auf 4096er-Schlüssel, die mit SHA-512 signiert sind aufzurüsten. Die neuen Fingerabdrücke können oben gefunden werden. 😉
  • Wir werden alle Mitglieder kontaktieren, die in den letzten zwei Jahren aktive Server-Zertifikate benutzt haben.
  • Wir werden neue Informationen an dieser Stelle veröffentlichen.

CAcert server move completed on December 12, 2013

The move of all existing CAcert servers  to a new smaller rack at the current hosting centre has completed, mostly successful, on December 12 at 23:00 UTC. All main services are available again now, but we still have some smaller problems to sort out, mostly due to the switch-over to a new much more compact firewall with a completely new architecture.

So please bear with us while we iron out the remaining problems, and feel free to report any issues you are still encountering.

Please join me in expressing thanks to the team that worked very hard over four hours after an already stressful day to get this major job completed: Stefan Kooman, Mendel Mobach, Martin Simons.

Blog update

The CAcert Blog (incl. its system) has been upgraded to recent software versions. With this also a new Theme had to be created to be compatible with the current version of the blog software.

Please let me know, if anything was broken by the upgrade or anything does not work for you as expected.

Maintenance / Changes infrastructure

[Update: finished.] I will begin with the work now 2012-04-01 00:00 UTC. Infrastructure e.g. email, wiki, blog, lists will be unavailable during the time. The CAcert website along with all critical systems are not “infrastructure” and will not be affected.

I am planning to work on infrastructure again this weekend (2012-03-31) and downtimes may occur. Our host is low on disk space so I need to do some restructuring. A larger downtime may be involved.

Currently, each virtual host has its own logical volume (LVM). They will be consolidated into one. During the move and resizing I think I will not get around shutting down all infrastructure services.

I will send out a notice before I begin the work.
Sorry for any inconvenience.

—-

I am currently continuing the work until 2012-03-26 07:00 UTC. The following services may be effected (the other services have been migrated already):

  • email
  • lists
  • emailout
  • board
  • translingo

I am working on infrastructure systems. During the next 6 hours the following services may be down as they are moved to another server:

* bugs
* irc
* webmail
* wiki
* email
* cats
* lists
* issue
* emailout

The following services have been shut down:

* dupes
* forum
* logging
* paypal
* test2

svn.cacert.org on new host with client certificate authentication now!

Today I finished the migration of svn.cacert.org to a LXC container on our new infrastructure machine. The container is running on Debian Squeeze and supports some nice new features:

Read only access is provided via http://svn.cacert.org/ as it was before.

Besides allowing client certificate authentication for our Subversion repository this is a big step forward as we now have a modern infrastructure machine with a recent operating system distribution.

If you already have a SVN account on svn.cacert.org and want to use the client certificate authentication feature please send a mail to svn-admin (at) cacert (dot) org.

The Big Masterplan to become Audit Ready

Back in January 2010 the former Board decided by Board motion m20100117.3 “No new subroots on current root, plan for new root”. In the discussion a date was scheduled by end of Dec 31, 2010. On my 2nd thought, probably nobody did recognize, what that means, CAcert's Big Masterplan To become Audit Ready (01/2010) to finish all the projects from the bottom left corner at beginning of 2010 to the top right corner by end of the year with the “New Roots and Escrow” (New Roots and Escrow) process running. So this article should bring Audits mistery to light.

Policy Group worked on the last few essential Policies (Policies on Policy Group), that are essential for the Audit. One essential requirement for Audit is to Rollout the CAcert Community Agreement to all the members, so they can decide to continue or to leave the Community. To become “CCA Rollout Ready” (CCA rollout), the running Software needs to be updated. This opens the next problem: by starting 2010, there was no Software Update Process defined, nor documented. But we’re on the lucky side, the Software-Assessment-Project started November last year to fulfill this requirement (Software-Assessment-Project). The task was: To get a repository system controlled by Software-Assessment team, a controlled testserver environment and a documentation system. Currently the team tests the transfer of a test patch to the production system. Involved parties: Software-Assessment Project team, Software-Assessment team and the Critical Sysadmins team.

CAcert's Big Masterplan To become Audit Ready (10/2010)
CAcert’s Big Masterplan To become Audit Ready (10/2010)

In the meantime, another issue pop’d up: the “Thawte points removal” with a deadline of Nov 16th, 2010. We’ve allready posted several blog posts on this topic. So also this is related onto the Software-Assessment-Project progress (Software-Assessment-Project).

The next topic is running Assurer Training Events (ATE) (Assurer Training Events). ATE’s are an essential concept in the Audit over Assurance (RA) business area. To scale a worldwide community, the community has to assist Auditors work in doing Co-Audits over Assurers. The question: How to contact groups of Assurers was answered back in 2009 with the ATE concept. The purpose of ATE is twofolded: first to communicate to the Assurers all the new informations and second to do Co-Audits. As Assurers follows the invitations to the ATEs we can expect, that they are more active in the community. So also from 2009 ATE experiences, we’ve got new resources from the community by contacts on ATEs (Get new resources). So this was the plan for 2010 ATE season, to find more people, who can help on the several tasks and projects that needs to be finished, before the new Roots and Escrow project and also the Audit can be (re-)started. E.g.

Helping CAcert

  • we are searching Infrastructure Admins for the Non-Critical Infrastructure systems, all running on Unix. Familiar with system migrations for the big Infrastructure project to separate Non-Critical from the Critical systems (The big Infrastructure Task). This project is running about 2 years, but currently without progress.
  • we are searching for Software Developers (C++, Python, Java) for the New Software project BirdShack (New Software Project BirdShack), that was started last year, after Auditors review of the Software that concludes: „Serious difficulties in maintaining, improving and securing.” and „Cannot form conclusion over software.”, so if the plan to start with the Audit over the old Software fails, we’re close to the 2nd path: BirdShack.
  • we are searching for Audit consultants who can assists in the Audit next step CrowdIt disclosure system (read AGM – Audit Report 2010 – CrowdIt. CrowdIt, as a sort of wordplay on Crowd-Audit). CrowdIt is an emerging disclosure tool (based on the old DRC browser).
  • we are searching people, who can assist us in the funding project (Funding project), that becomes the ground base for the New Roots and Escrow project (New Roots and Escrow) that should be keep tracked by an Auditor, and the re-start of the Audit (Audit over Assurance (RA) 1) and (Audit over Systems (CA) 2).

The New Roots and Escrow Project Relation to Audit

As said before, the New Roots and Escrow Project should be keep tracked by an Auditor. From the experiences back in 2008 on creating New Roots but fail on Roots Escrow, we’re warned to separate the Audit steps of the New Roots and Escrow Project (New Roots and Escrow) and the Audit over Systems (Audit over Systems (CA) 2). Both tasks should be close together.

On the other side, we have to do an Audit over Assurance (Registration Authority, RA) (Audit over Assurance (RA) 1). There is no requirement on bundling the RA Audit and CA Audit as both business areas have their own Policy sets and can be checked separately. This can make our work presumably easier. Easier to get Audit funding for Audit over RA. As Assurance area is closer to be Audit Ready, we can also signal to the Community Audit is back on track. This will probably push the other tasks. With a small budget we probably can double the result by getting new resources, “Hey, there is progress on the overall Audit task” – CAcert is back!