Improving Distro tracker to better support Debian teams
Name: Arthur de Moura Del Esposte
Affiliation: Masters student in Computer Science at University of São Paulo - USP
Basic Information
Email: arthurmde@gmail.com / esposte@ime.usp.br
Nationality: Brazilian
Applying for other projects in SoC: No
IRC: available as arthurmde in #debian-qa and #debian-devel-br on OFTC
Github: https://github.com/arthurmde
Gitlab: https://gitlab.com/arthurmde
Salsa-Debian: https://salsa.debian.org/arthurmde-guest
Background
I have a bachelor's degree in Software Engineering from the University of Brasilia (UnB). During my graduation period, I became a free software enthusiast, and since 2013 I have attended community events and contributed to free software projects. My early involvement with these communities contributed significantly to my education, both in the technical and social aspects, as well as bringing me many opportunities. Among others, I highlight the opportunities to become a member of a research lab in FLOSS and agile methods at UnB, to work as a developer on the Brazilian Public Software Portal project, and to work with great professionals from important free software projects, such as Debian and Mozilla. Currently, I am pursuing a master’s degree at the University of São Paulo (USP). In USP, I belong to the FLOSS Competence Center. During my masters, I have been developing the InterSCity platform, a microservices-based, cloud-native middleware for smart cities, which is the main free software project that I’ve worked on recently.
Skills
As a consequence of my training and my earlier contributions to free software projects, I’ve developed various technical skills which include good practices of software engineering, TDD, refactoring, system design, microservices architectures, and ?DevOps.
I work with Linux, Git, and Vim daily. As most of the projects I've worked on is Web-based, I’m quite familiar with web technologies such as HTML, CSS, Javascript, Bootstrap, ?WebSocket, and web frameworks. More precisely, Ruby is my primary programming language, and I have three years of experience with Ruby on Rails. Also, I’m fairly confident to work with Python and Django since I used them before in other projects. Finally, as I became more familiar with microservices architecture, I’ve been working with ?DevOps tools such as Docker, Kubernetes, and Ansible.
Project Title: Improving distro-tracker to better support Debian Teams
Synopsis: This project aims at designing and implementing new features in Distro Tracker to better support Debian teams to track the health of their packages and to prioritize their work efforts.
Project Details:
Distro Tracker concentrates several useful features related to Debian packages, allowing teams to group it packages of interest and to receive update notifications from it. Although Distro Tracker provides a comprehensive page with detailed data for each package, it lacks features for helping packaging teams to have an overview of their packages.
Currently, the PET (Package Entropy Tracker) project implements useful features for packaging teams tracking the status of their packages, providing useful information about upstream projects, and supporting QA tasks by highlighting packages with bugs, missing tags, and outdated. However, PET is no longer active and is obsolete as it does not integrate the data from Salsa development server. Therefore, the idea of this project is to migrate the most important team-related features from PET to Distro Tracker, leveraging and improving Distro Tracker current code base regarding teams. Thus, as a final result of SoC, I expect to incorporate to Distro Tracker a set of useful data to help teams to see the health of multiple packages and better prioritize their efforts where it is most needed. It is worthing noticing that Distro Tracker is a general purpose service that is also used by Kali community. Thus, they also will be able to take advantage of the proposed improvements.
Benefits to Debian:
Distro Tracker is a key service of Debian infrastructure. Debian packaging teams will benefit from this project since they will be able to visualize better and organize teamwork, tracking the packages and prioritizing the team efforts. Besides, after migrating the main features from PET to Distro Tracker, it will no longer be necessary to keep PET running on Debian’s infrastructure.
Deliverables / Final Aim:
Among other features, I expect to design and implement the following:
- New background tasks to import data from Salsa and integrate it with the current data available on Distro Tracker
- Improve current team page to show summaries of the most relevant information from a team's packages
- Implement additional pages with the details of this package based on PET information flaps
- Provide interactive search of team packages with filters and pagination
By the end of the SoC program, I expect to have these features incorporated into Distro Tracker official repository and running on the production environment in [http://tracker.debian.org/] Finally, I will write a series of blog posts presenting and discussing the contributions in each phase of the project.
Project schedule:
It is difficult to pinpoint the time required to perform each task, mainly because there is an uncontrollable patch period and patch fixes sent to the main repository. Besides, these estimates may evolve from the advances that are being made and agreed upon with the mentor.
04/23 (Weeks 1, 2, and 3): Experiments with Gitlab and Debian Archives
- Learn more about Debian packages and related infrastructure
- Learn the concepts behind Debian Archives
- Learn Gitlab API and make experiments with python-gitlab package
- Learn Gitlab Webhook specification
- Define the data that will be imported into the distro tracker and design the improvements that will be implemented in the models and tasks of the Distro Tracker.
- Propose and discuss with the community changes on model classes to save imported data
- Send general contributions to Distro Tracker, mainly focus on fix all bugs related to teams and packages features.
05/14 (Week 4) - Coding officially begins!: Task to import data from Salsa (Gitlab)
- Create task to import data from Salsa
- Create parser for Gitlab API
- Implement the proposed changes on model classes to save imported data and add new classes when required
05/21 (Week 5): Improve the task that imports data from Debian Archives and Add missing data to packages.
- Improve data structures if needed
- Import missing data if needed
- Set the following additional data for each package to support low-latency queries
- Packages with RC bugs - From BTS
- New packages - Archive
- With bugs - From BTS
05/28 (Weeks 6 and 7): Add useful data used by PET to packages
- Monitor Debian Changelog on Salsa to mark packages as Ready for Upload
- Add Missing tags flag to packages by comparing data from Salsa and Archive
- Import data from Debian Watch to define the following information for each package:
- Newer upstream version
- Problems with Debian/Watch
- Define package as Work in progress by monitoring Salsa repository
- Submit code and collect feedback from the community
06/11 (Week 8): Add new metrics to Package page
- Discuss how to present the computed metrics in package page. A good approach could be to use medals to demonstrate the current state of a package.
- Implement the defined approach in Package page
06/15 Midterm evaluation (Submitting Phase 1)
06/18 (Week 9) Improve team page to present summaries of most relevant package information of a team
- Add proper database index to improve performance of queries to teams packages
- Improve team page with the summary of each collected metric. We can show them through charts or more simple visualization mechanism
- Submit code and collect feedback from the community
06/25 (Week 10): Create individual templates for each metric
- Create individual templates for packages marked as:
- Newer upstream version
- Problems with Debian/Watch
- Work in Progress
- Missing tags
- Ready for Upload
- With RC bugs
- New package
- With bugs
- Create routes to individual pages for each metric
- Include templates to the main page with a show/hide behavior
- Submit code and collect feedback from the community
- Create individual templates for packages marked as:
07/02 (Weeks 11 and 12): Add search feature to team page. Final contributions and improvements.
- Add search to facilitate teams to find their packages
- Submit code and collect feedback from the community
- Fix previous patches reviewed by community members.
07/13 Final Term evaluation (Phase 2 Evaluation deadline)
07/16 (Weeks 13 and 14):
- Any final works required by the maintainers and by the mentor.
- Post about Final result of the project
Present my project at DebConf18 in Taiwan (depends on travel funding)
Contributions to Debian
I am a Debian user for years, but I started my contact with the Debian community way before that. During my graduation, I helped in the accomplishment of a micro event that gathered four Debian Developers in Brasília - Brazil. Also, in this same event, I participated in a packaging tutorial where I did QA fixes in the ajaxterm package.
Recently, I have been worked on the Distro Tracker making a number of contributions to the source code. The following list details all the accepted patches I sent:
- 21735574a - Removing priority and section fields from Source Package
- 93436a8 - Add a new Mixin with helper methods for testing the existence of objects
f3663f - Add ?UserAuthMixin with helper methods to manage user authentication
98f2c5f - Add tests for ?CreateTeamView
b3defe7 - Add tests for ?TeamDetailsView
885a10 - Add tests for ?DeleteTeamView
b2112ad - Add tests for ?UpdateTeamView
97f1ab58 - Add tests for ?AddPackageToTeamView
b6f17aa1 - Add tests for ?RemovePackageFromTeamView
94a3364e7 - Add tests for ?JoinTeamView
61aa85cb - Add tests for ?LeaveTeamView
c1ebc1dc - Add tests for ?ManageTeamMembers
dd58c48f - Add tests for ?RemoveTeamMember
cb1546269e - Add tests for ?AddTeamMember
7d2686cf - Add tests for ?ConfirmMembershipView
56aaa64 - Removing unused template and return statement in ?ConfirmMembershipView
46aec77b - Add tests for ?TeamListView
8003414 - Add tests for ?SetMuteTeamView
f6b1f7c - Add tests for ?SetMembershipKeywords
3260e6a - Add tests for ?EditMembershipView
- 50c07f4 - Introduce reusable partial template to show profile messages
- 89560cf - Make news' url more human friendly by using its title slug
- 6a7c9d - Accept trailing slash on news URL
- 6bc29b2 - Improving autoremoval description by adding buggy dependencies links
Most of the contributions are related to the code modules that I will be working in during the SoC, which includes Team, Team Member and Package related models and views. The above listed contributions resolved the following bugs:
Add unit and functional tests for features related to a team
Server Error (500) when one add an existing member to a team
Tracker news URLs could be more package related + human friendly
Autoremovals: display the package that is triggering the removal
Finally, I am going to attend Curitiba’s MiniDebCamp and MiniDebConf between April 11 and 14. The two first days I will work with the local community on new contributions to Debian, both to Distro Tracker and packaging, during Bug Squashing Party and the sprints that will happen there. The last two days lectures, debates, and more collaborative work. This will be an excellent opportunity to strengthen my relationships with the Debian community, meet other people involved, and learn more about the project, serving as a "warm-up" for the SoC as well.
Other Notable Contributions
Since my first contacts with free software, I started to contribute to the source code of the projects. My participation ranges from small bug fixes in utility tools to contributions in key features of some projects in which I work more actively. The most notable projects to which I contribute are listed below:
InterSCity - this is a middleware platform for smart cities based on microservices architecture. I am the main committer and maintainer of this project as it is a direct outcome of my master’s research. As I am working on it from the beginning, I was able to learn about several aspects of building and to organize a free software community to enable external contributions. The project has already had several external contributors, but its community is growing by now as other research groups have started to use it.
Brazilian Public Software Portal (SPB) - SPB is an important project from Brazilian government which aims to boost the adoption and sharing of free software among various sectors of the government. I worked in SPB as a software engineer for almost two years, contributing actively to the project and working with members of other free software communities.
Noosfero - Noosfero is a web platform for social and solidarity economy networks with blogs, e-Portfolios, CMS, RSS, thematic discussion, events agenda and collective intelligence for solidarity economy. In this project, I worked on several plugins and new features, in addition, to help with the system migration from Ruby 2 to Ruby 3.
Radar Parlamentar - a web app that illustrates the similarities between political parties based on the mathematical analysis of voting data that occur in the legislative house. This is a Python/Django project for which I contribute during its earlier stages.
Other Information
Exams and other commitments: No other commitments during the GSoC.
Where will you work: Mostly in USP campus and sometimes in my home.
Other Summer Plans: None
Why Debian?: I have been using Debian for years. I admire the project for its technical quality, for being a project that prioritizes its users, and for the philosophy and organization of the project community. I am very keen to contribute back to the project actively, I believe that I have technical skills to assist the project in several aspects, but I also see it as a precious space of much learning and knowledge. SoC is an excellent opportunity for me to dedicate to the project and to participate in the community. I intend to continue contributing and getting involved in other aspects of the project after SoC as well as envisioning the opportunity to become a Debian Developer in the future.
Are you applying for other projects in SoC?: No