Page MenuHomeMy privacy DNS

Contributing to the web frontend?
Closed, ResolvedPublic

Description

Hi @DRSDavidSoft To continue our talk from https://github.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/issues/590 regarding your web coding in PHP.

To open the question, then it's about getting a web interface up and running from where users can submit/report any domains. With this project Matrix and my question is at first, could you be interested in contributing with such an interface, from where different actions can be done?

A few rough ideas is:

  1. Users can add a domain or URI, which then is opening an issue on this site and generating a PR in the proper repository on (GH).
    1. This will involve the ability for users to set there GH API key, in a profile
  2. An interface from where ADMIN's can accept or approve a contribution.
    1. If approved commit PR AND add or delete the corresponding records to the DNS server (API)
  3. As @funilrys and I is working on getting a central database up and running for the #pyfunceble project written and maintained by @funilrys we could use some nifty tools
    1. Get the data from ad 1 into the test flow
    2. A interface for submitting users own list, and then post the results from the #pyfunceble_-_cluster_-_master
    3. An interface for users to extract data in various formats like RPZ record types
  4. A domain categorizing and maintaining system. Example: pornhub.com is Adult Contents and google is Spyware, Adware etc. etc.

All users should be signed in/up through the GH API to have them fiddle with the GDPR and user data administration/troubles.

This is the rough scratch of the question to you, could it be something for you to fiddle with in your spare time?

Best regards @Spirillen

Event Timeline

Spirillen triaged this task as Wishlist priority.Sep 7 2020, 4:38 PM
Spirillen created this task.
AnonymousPoster shifted this object from the S1 Default Global Public space to the Restricted Space space.Oct 25 2020, 6:48 PM
Spirillen shifted this object from the Restricted Space space to the S1 Default Global Public space.Feb 17 2021, 10:50 PM

Nice ideas overall! Hopefully, all can be implemented in time with contributions from me and others.

In the coming months, I'll be busy as I'm working on a high priority project, but I have a couple of notes that I'd like to contribute in the meantime:

  • There are several similar services to take inspiration from, e.g. DeveloperDan's Blocklist Tools, Hell.sh's Collective Hosts File, and oisd project
  • I've been privately working on a similar project of my own, named Openlists, since I was unsatisfied with existing projects. (the code is not yet pushed to the repo)
  • The way I write a web-based app stack, the http API should be accessible to both web-based front-end (ajax/async method) and a terminal-based (cli) query utility (e.g. dnsbl --query google.com), more explained below
  • I have already written some scripts for managing my backend:
    • scan all sources, parse domains, add everything to a central database
    • only unique new domains are added and each is assigned an id
      • this means that lists are automatically deduplicated, and I can see which domain is exactly present in which list
      • also, I can see what percentage some lists have in common (also useful for sorting out aggregated lists)
    • ability to declare mirror URLs for the same source, detect dead URLs, etc
    • a domain can be queried by the user in the following format:
      • a.b.c.d can match itself exactly, or through b.c.d and c.d domains
      • a.b.c.d can point to a CNAME, which may be blocked
    • a user can see if the domain should be blacklisted, or whitelisted
      • comments are parsed and displayed to the user, so they can understand why a domain is put into each list
    • the variations of domains (such as www.xxx.tldxxx.tld, etc) are detected and corrected
    • if b.c.d covers a.b.c.d, then the latter is ignored when generating the output
    • if several subdomains are blocked but the parent domain is not, suggest to also put the parent domain into the list (e.g. 1.x.y.z, 2.x.y.z, and 3.x.y.z is blocked, but x.y.z itself is not)
    • ability to query the timestamp of when a domain entry was added or removed to each list
    • analyze the frequency and number of the blocked domain that are hit
      • ability to suggest certain sources to include or exclude
      • ability to filter out dead/unused domains

When I have time to contribute, I'd like to both contribute to code for front-end and back-end. Also, I'd like to know what kind of design you have in mind.
Regarding the back-end, you are already using pyfunceble's great features, which is quite nice.

Personally, I use PHP's built-in features to query domains through a DNS server in my scripts.

If there's no rush, I'd be glad to help in any way I can, although at the moment I can only assign small amounts of time, due to being committed to a project that I'm already working on. If you'd like, I think having a centralized database to store all domains that have ever existed in different sources is also a good idea. I'd also be glad if I can work on reusable components that can be shared across projects (kind of like how pyfunceble is used in many different projects).

Also, I'd like to also stress out that I believe providing a commandline script - that can be used in the same way a web-based front end can be - is essential, since many of the users might be power users that prefer it to loading the page in a browser. This requires providing a public API, and isolating the front end from the back end (only using ajax/asyc queries, instead of rendering the HTML page content directly from the PHP scripts).

This seems to be a great project, I only wish I can have enough time to work on it appropriately.

I've written down my idea to the search section:
Everything with # is a comment for design. Below is the "magic" new frame which opens on click on a version.

mockup_matrix_search-ui.jpg (3×2 px, 1 MB)

Hey @DRSDavidSoft

Your reply have totally slipped my notification 😭 and you should absolutely have had a response earlier.

I'm in general happy for your suggestion, but will stick to the #pyfunceble as the backend for testing and storing the data.
I would like to ping @funilrys to this thread, as how things are going to be done in the backend is more into his hands and minds.

@Somebodyisnobody I like your suggestion 👍🏻