Human-centred design (HCD) approaches are adopted to develop digital agriculture interventions inclusively and responsibly. Whether these approaches indeed lead to responsible designs remains unclear, especially for low-income countries. Using a Rwandan case-study, we contribute to debates on inclusive, participatory and responsible design by developing a framework for operationalising HCD and responsible innovation in practice and studying the process of designing a digital agriculture intervention for banana disease management. The four dimensions (inclusion, anticipation, reflexivity, responsiveness) of responsible innovation and our own framework of digital rights served as analytical lenses. Findings show that power relations and digital capacity negatively affect user inclusivity in design. The context in which HCD is deployed hinders anticipation, reflexivity and responsiveness, resulting in design decisions that do not fully respect digital rights and thus in potentially irresponsible digital technologies. Broader, long-term consequences of digital technologies should be a central consideration in design processes, while responsible innovation theory needs to become cognizant of the complex realities in which digital innovations emerge.