Page MenuHomeMy privacy DNS

Matrix infrastructure by @Somebodyisnobody
Closed, ResolvedPublic

Description

Related to T1901 I also want to make a suggestion how Matrix looks for me.

I've created a general approach which is based on modules.

Matrix overview @Somebodyisnobody (2ร—7 px, 1011 KB)

  • We have the frontend which is used by some stakeholders. Those are communicating with the web frontend (in the end with the web backend) or with the web backend REST-API.
  • The backend is the gate to the database and processes any logic that's based on direct interaction with Matrix. This means that the user will never communicate directly with pyfunceble.
  • The database itself is relational in this approach as I had no better idea. For DB-concept of the versiong system please have a look into T2686#58532. In the overview there's just one database which doesn't mean that we cannot have a separate DBMS for account management or a Redis DB for caching.
  • The asset storage is a central place where generated lists stored that are ready for distributing. Assets are then retrieved by the backend on demand. or by other Matrix-related services like the DNS-servers
  • As I learned, Pyfunceble is only for checking the availability of domains. This causes a need for another aplication which cares about provisioning the local services, generating assets like RPZ-rules etc. This work could also be done by the web backend but to keep it modular I decided to draw a worker. This worker has an API which the web backend is communicating to. As long as we have only one worker, he can also to the iterating jobs.
  • Pyfunceble does the checks for active domains It reports back wheth the collected results about which domains are dead and which cname-mappings (exemptions) are live. This data can be processed in later business products. Inactive domains can be committed over the web backend into the database and then distributed to other projects via web backend.
  • The DNS auth server gets the RPZ-rules by the worker. Consumers can then download the RPZ-rules via DNS after they saw that the serial number of the zone changed.
  • The DNS-recursor (not on the picture) isn't what I cared about long time but with T1900#58535 in mind I strongly recommend to let only the backend talk with the DNS-server to set ACLs.

The system can be monitored by Icinga and logs can be collected and aggregated with Graylog on Elasticsearch.

For the web backend I recommend SpringBoot (Java framework for web applications) as it has a running instance which results in faster processing and lower query time on the API-side. The web backend can be split into retrieving data from asset storage, backend api and frontend api. This makes maintenance and downtime-handling easier. For the worker I'm not sure if we shall rely on SpringBoot or Python. I've no experience with Python. A plus to Springboot is that it delivers a built in scheduler which would be nice for periodically iterating jobs.

Event Timeline

I thought about doing the generation of personalized files in the web backend but this would mean that we have to code logic twice which is already done in pyfunceble. So the workflow is: web backend bumping pyfunceble to generate a custom personalized list in the target format, pyfunceble hooks the web backend when it's finished and stores the asset in the asset storage where it can be provided by the backend until it expires.
This doesn't affect DLEF-files (T3776) as they represent the unprocessed "stock dataset" with blacklisted or whitelisted entries.

Like having a view into my own mind ๐Ÿ˜„

However I do have a few questions to your suggestions.

  1. When you say DNS-server Could I request you to split this into the processes of DNS-AUTH, DNS-recursion or Dnsdist as they are very different processes.
  1. Icinga and SpringBoot, as long logging NEVER collects any user data, I can work with it, and it is ONLY used for back end management ๐Ÿ‘๐Ÿป (Both are new to me)

ping @funilrys, what do you think?

  1. When you say DNS-server Could I request you to split this into the processes of DNS-AUTH, DNS-recursion or Dnsdist as they are very different processes.

Okay but you need to explain me the difference first via pn xD. I have a weak idea what's the difference and brooding where/why a DNS-Auth is running in Matrix

  1. Icinga and SpringBoot, as long logging NEVER collects any user data, I can work with it, and it is ONLY used for back end management ๐Ÿ‘๐Ÿป (Both are new to me)

Icinga would do periodically http requests to the web stuff and alerts when a 500 is thrown for example. SpringBoot-logging can be just not implemented and I intended Graylog/Elasticsearch (as logging tool for pyfunceble generator logs, failed logins on the backend, thrown >400 status codes ecetera)

I just updated the overview. @Spirillen can you play with the configs to get Phabricator to render this?

I can't do anything against this, I tried a few days back.

This image (with dimensions 10,000px x 3,000px) is too large to transform. The image has 30,000,000 pixels, but transforms are limited to images with 16,777,216 or fewer pixels.

You may get a smaller screen like the rest of us ๐Ÿ˜„

Looks good to me.

Just keep in mind: PyFunceble does not have to be necessarily a script. I'm working on a project which will just provide PyFunceble but behind a web API. Meaning that you could literally do everything in Java and ask PyFunceble (or its underlying infrastructure) about a subject.

What is meant with "its underlying infrastructure": PyFunceble will at some point also have an API/Service that collects the "live" results from trusted instances from around the world. Meaning that you could literally ask that API/Service if you have something about the subject in the last X hours and go ahead with what you get before even launching a PyFunceble test.

I'm wondering if we shouldn't split everything related to pyfunceble with everything you propose and join them with the worker. ๐Ÿค”


For the worker I'm not sure if we shall rely on SpringBoot or Python.

Well, it depends on what is explicitly meant behind it. But it can be either of them. A fast API application along with celery does the same.

Actually, the tool/language in a WEB backend which acts as a DB gateway is not really that significant. It can be one or another.


In the overview, there's just one database which doesn't mean that we cannot have a separate DBMS for account management or a Redis DB for caching.

Maybe an 0Auth2 compatible endpoint and a microservice/backend on their other side which acts as a "global" Identity provider.

Why an identity provider? The earlier this side is clearly defined the better it will be to integrate, manage, or extends future services.

-------------

To keep it as easy as possible we should keep this in mind:

  • Source code controlled with git.
  • Databases SHOULD be either a MariaDB or PostgreSQL one.
  • Database migration controlled with either alembic/sqlalchemy (python) or their Java equivalent (Ever used Flyway @Somebodyisnobody ?)
  • (idea) Database should be replicated to avoid downtime. And/OR adopt a canary, blue-green, or red-black (for the paranoid) deployment pattern.
  • (idea) Think about a develop/staging env.

Hey @funilrys and @Somebodyisnobody

My feedbacks to this, seen from the perspective of maintenance, I would clearly vote for MariaDB as the sql backend(s) and more DB's is better as we can distribute over more resources,

(idea) Database should be replicated to avoid downtime. And/OR adopt a canary, blue-green, or red-black (for the paranoid)

You can easily strike the (idea), it is a most, just as it is a most to get some โ‚ฌโ‚ฌ to make it possible and where this is currently standed, however, I do actually fetch hourly to my own server at home so the cab is minimal ๐Ÿ˜„

Spirillen triaged this task as Wishlist priority.Feb 26 2021, 12:01 AM

Just keep in mind: PyFunceble does not have to be necessarily a script. I'm working on a project which will just provide PyFunceble but behind a web API. Meaning that you could literally do everything in Java and ask PyFunceble (or its underlying infrastructure) about a subject.

What is meant with "its underlying infrastructure": PyFunceble will at some point also have an API/Service that collects the "live" results from trusted instances from around the world. Meaning that you could literally ask that API/Service if you have something about the subject in the last X hours and go ahead with what you get before even launching a PyFunceble test.

that's nice, I'll come back to that point when talking about the worker.

I'm wondering if we shouldn't split everything related to pyfunceble with everything you propose and join them with the worker. ๐Ÿค”

what would that "everything related" be? The worker in my example would remove dead entries from the matrix-blocklist. Do you mean that

Why an identity provider? The earlier this side is clearly defined the better it will be to integrate, manage, or extends future services.

Do you have a software solution for that?

To keep it as easy as possible we should keep this in mind:

  • Databases SHOULD be either a MariaDB or PostgreSQL one.

I heard that PostgreSQL does not support partial indexing. If that's right we should use MariaDB but let's check that later.

  • Database migration controlled with either alembic/sqlalchemy (python) or their Java equivalent (Ever used Flyway @Somebodyisnobody ?)

nope will look for that when the first stable database system is going to be deployed to iat.

  • (idea) Think about a develop/staging env.

Yeah I'll setup an iat at home and we can set up one later in the web.

@Spirillen can you tell me a way how to categorize tasks for the modules (asset server, web backend, database etc.) in matrix.rocks on phab?

The worker in my example would remove dead entries from the matrix-blocklist

This is, as such, (un)-decidable, as it can be a temporary fail-out, however we (funilrys and I) have been talking about adding a "First time seen dead" so we can add a trusted delayed delete cycle. ie. seen dead for 60days -> delete

That would also add trust to blacklist maintainer for when it is safe to automatically delete any record from there source's.

... how to categorize tasks ...

Do you mean #tags (Projects) ?? let's take that one on keybase.

The worker in my example would remove dead entries from the matrix-blocklist

This is, as such, (un)-decidable, as it can be a temporary fail-out, however we (funilrys and I) have been talking about adding a "First time seen dead" so we can add a trusted delayed delete cycle. ie. seen dead for 60days -> delete

That would also add trust to blacklist maintainer for when it is safe to automatically delete any record from there source's.

Yeah if pyfunceble holds this information then I don't need to code it duplicated. ๐Ÿ‘Œ
Does pyfunceble also check the registration status of domains?

Yeah if pyfunceble holds this information then I don't need to code it duplicated. ๐Ÿ‘Œ
Does pyfunceble also check the registration status of domains?

It does it's best through the WHOIS system, but since not all eTLD's maintain a WHOIS server, it can not be complete. This is also here my suggestion for a independent db.domains are coming into play again as is should be the "Glue" DB between sub-routines / Sub-processes