Open-Source Data Lake Platform for the Secretariat Digital HUB

This RFI is issued solely for market research and information-gathering purposes. It does not constitute a solicitation for proposals, bids, or quotes, nor does it commit UNGSC to pay for any costs incurred in responding. Responses will not be evaluated for selection or award, and no contractual obligations will arise from this RFI. Information provided will help UNGSC understand available open-source-based data lake platforms and related services in the market and contribute to developing a scope of requirements for a potential future Request for Proposal (RFP). All responses will be treated as confidential where indicated by respondents.

Background

The Infrastructure and Operations Section of the Service for Geospatial Information and Telecommunication Technology (SGITT), United Nations Global Service Centre (UNGSC), Department of Operational Support, installs, operates and maintains Information and Communications Technology (ICT) systems for the United Nations Secretariat, including the Secretariat Digital Hub.

The Secretariat generates and collects growing volumes of structured and unstructured data from databases, applications, files, logs and streaming sources. A data lake platform is required to ingest, store, process, analyse and share this data in a secure environment that remains under the full control of the Organization, in line with its data protection and information security requirements.

The Organization has assessed consumption-priced proprietary cloud analytics services, whose capacity-based charges recur every month for the duration of use. It intends instead to deploy a modular platform based on open-source software on UN-managed infrastructure, supplied, integrated and supported by an external contractor, and to retain ownership of the resulting configuration and know-how.

UNGSC is issuing this RFI to gather information on open-source-based data lake platforms, and on the design, integration, support and training services offered around them, available in the market. Interested suppliers are invited to provide detailed information on their solution’s components, architecture, security, integration capabilities, implementation approach, licensing and support services. This information will inform UNGSC’s understanding of commercial options and support the development of requirements for the Request for Proposal (RFP) for an open-source data lake platform for the Secretariat Digital Hub.

Key areas of interest include

• Modular Structure: The requirement is envisaged in three lots:

A. Core Platform (design, deployment and integration),

B. Object Storage (optional), and

C. AI/ML Services (optional), together with support and training services.

• Open Source Without Recurring Charges: All platform components under OSI-approved open-source licences, with a software bill of materials, and no paid licence, subscription or capacity fee required to run them.

• Deployment on UN-Managed Infrastructure: Containerised deployment on UN-provided servers, storage and network, on a cluster of at least five nodes, with no UN data leaving UN-managed infrastructure.

• Security and Governance: Single sign-on, role-based access control, secrets management, security monitoring, encryption, hardening and audit logging, aligned with UN Secretariat information security and data protection requirements.

• Ownership and Knowledge Transfer: Scripts, configurations, infrastructure code, pipelines and documentation becoming the property of the Organization, with knowledge transfer to UNGSC IT staff and handover at the end of the contract.

• Support and Service Levels: An annual support and maintenance subscription with defined service levels, and the ability to deliver on-site services at both Brindisi and Valencia as well as remotely.

Information Requested

Suppliers are encouraged to address the following sections in their responses, providing examples, documentation, or references where possible. Responses should focus on how their solution meets or approaches these areas, including any relevant case studies from similar organizations.

A. Functional Fit

• Data ingestion and streaming: flow-based ingestion with flow versioning, event streaming, and connectors for relational databases, REST/SOAP APIs, files, logs and CDC sources, with data lineage (reference: Apache NiFi, NiFi Registry, Apache Kafka or equivalent).

• Storage: distributed file storage and relational database, open file and table formats, with backup and recovery (reference: HDFS, PostgreSQL, Parquet, ORC, Avro, Iceberg or equivalent).

• Processing, orchestration and query: batch and stream processing, distributed SQL engine and workflow orchestration (reference: Apache Spark, Trino, Apache Airflow or equivalent).

• Analytics and BI: dashboards, multi-user notebooks, SQL exploration and export of results (reference: Apache Superset, JupyterHub, Apache Zeppelin, Hue or equivalent).

• Optional modules: object storage (reference: MinIO, Ceph, Apache Ozone or equivalent) and AI/ML services (reference: Ollama, Open WebUI, MLflow, vector databases or equivalent).

B. Technical & Integration Capabilities

• Architecture and deployment model, including container orchestration on a cluster of at least five nodes (reference: Docker Swarm or equivalent), and the infrastructure sizing required (cores, memory, storage, network).

• Gateway and platform operations: reverse proxy, monitoring, log management, source control and CI/CD (reference: Nginx, Grafana, Graylog, OpenSearch, GitLab, Jenkins or equivalent).

• Integration with UN identity, monitoring, logging and CI/CD services.

• Open-source components used, versions, upstream communities and licences.

C. Usability & Adoption

• Quality of UX/UI for data engineers, analysts and administrators.

• Accessibility and self-service capabilities.

• Learning curve, documentation, and training materials for administrators, data engineers and analysts.

D. Security & Compliance

• Single sign-on (OIDC/SAML), multi-factor authentication and role-based access control (reference: Keycloak or equivalent).

• Secrets management and security monitoring (reference: Passbolt, Wazuh or equivalent).

• Encryption in transit and at rest, hardening to a recognised benchmark (e.g. CIS) and audit logging.

• Vulnerability and supply-chain management for open-source components.

• Alignment with UN security policies and standards.

E. Implementation & Support

• Implementation methodology and indicative timeline, from design and project plan through deployment, pilot data pipelines, testing and acceptance, to documentation and training.

• Annual support and maintenance subscription: patching, upgrades, security advisories, incident handling, and proposed service levels and service credits.

• Support and training services: professional consultancy, platform administration training (online), on-site training and knowledge transfer, and remote and on-site support.

• End-of-contract handover and orderly transition.

• Experience and references in the design, deployment and support of production open-source data lake or big-data platforms.

F. Cost & Licensing

• Open-source licensing and any optional commercial support subscriptions for upstream components.

• Pricing structure for each lot, the annual support subscription, and support and training services.

• Total cost of ownership (TCO) over a three to five-year period, and flexibility in scaling capacity.

Annex A – Market Information Questionnaire provides further questions to support market research on the proposed platform. Respondents are encouraged to address these questions in their PDF response, identifying relevant open-source components, commercial add-ons, alternative approaches and any important limitations. The questions are intended to guide information gathering and are not evaluation or pass/fail criteria.

Response Instructions

• Format: Submit responses in PDF format, not exceedi