OSPOlogyLive Malmö - Roundtable Meeting Notes (Day 1)

[anajsana]

Creating a thread for Roundtable leaders to share their meeting notes (leaving a comment) gathered during their roundtable sessions on day 1 (Thursday 30)

This Thread will later be linked to OSPOlogyLive Malmö Knowledge archive (roundtable section)

[anajsana]

ROUNDTABLE 1

Attendees coming from different companies, semi-public sector and academia, including Axis, Volvo, Ikea, BBC, etc

How to Define Objectives and set up meaningful metrics to be digested by C-level Managers?

• How to identify risks
• How to identify where to put our effort in
• How to save time (developer efficiency)
• How to talk to managers to show ways open source help from a business perspective
• Deal with different maturity levels of open source processes across teams and business units
• High-level questions coming from c-level managers: upstream contributions from employees, how much code are we open sourcing

Recap on the OSPO OKR's

• Different types of costs: Speed vs. license costs
• Have a long lifespan when looking into metrics (continuous monitoring) to avoid the "OSPO is a cost-centric" team by managers
  • The power of the OKRs aims to help this cost-centric mindset. With a common objective that leads to a long-term objective, be ambitious, and not focus on short-term goals
  • "Weight of Maintainance reduction" as a metric for developer efficiency

Brainstorm - Drivers of motivation:

• Costs (indirect adn direct effects)
  • e.g open source migration is a direct cost. From that, to try to match your business requirements
  • adoption or open source released tool as a framework
• unblockers within engineering teams
• Talent retention / Talent attraction
  • How can OSPOs help talent acquisition (e.g strategy to open source tool, to be implemented as a )

What decisions to make when you don't have data in place?

• How to measure something that is not even created (e.g incubating projects, starting Working Groups) to justify the funding/sponsorship

Software Supply Chain Sustainability Narrative to C-level Objectives

• Metrics around Technical Debt of open source projects
• E.g in FINOS Foundation use case: they first discuss the business impact, and then they collaborate together to open source a problem. Building confidence with competitors.
  • Identify commodity software
  • Create an ecosystem with your competitors, sharing development costs for the commodity software

[ccosborne]

ROUNDTABLE 2: What will it take for open models to be enterprise-ready?

• We began our group discussion with slides that provided an overview of 17 components of models identified in the Model Openness Framework, very few of which are typically released.

• We discussed why few components are released and the barriers to enterprise adoption of open models. For example, we discussed the misalignment and frictions between the open science principles of AI researchers and the interests and needs of companies. In most cases, proprietary data and models are too sensitive to share (e.g. "secret sauce", data protection, etc.). One participant explained that it will be rare for companies to comply with all requirements of the Model Openness Framework (i.e. release of all 17 components used to develop open models), and that it is more realistic that companies comply with the requirements of tiers 2 and 3.

• We discussed the security concerns of enterprise adoption of open models hosted on platforms like the Hugging Face Hub. Due to the limited information and components shared about models, even models released by industry leaders, it is very hard to audit models. One participant mentioned that it usually takes 3-4 weeks to conduct technical due diligence on new software products, which is insufficient to do thorough tests on poorly documented open models. Another participant mentioned that it is difficult to get external consultation due to the privacy of customer data.

• We discussed the lack of knowledge concerning the harmful impacts or potential harmful impacts of open models. One participant mentioned the role of institutional review boards in academia to evaluate new research for its innovation and risks, and these boards are currently lacking for the review of open model development.

• We also discussed the barriers to facilitating multi-vendor collaboration on open model development pre-release due to the proprietary constraints. Grassroots initiatives like BigScience, which led the development of the BLOOM large language model involving over 1,000 contributors from over 250 organisations, the support of the French government was critical for the development of the model by providing acess to its Jean Zay supercomputer. This example highlights the criticality of access to computing power to facilitate "open source AI" which is a major barrier to bottom-up development of open models. One participant suggested that if we want to see more grassroots initiatives, we will need more publicly-provided computing power. Two participants mentioned previous European projects to build federated cloud infrastructure. They are still trying to prove that it's possible, but it's a promising direction.

• We discussed whether OSPOs should develop policies for open model development and use. Participants agreed OSPOs should do something about it but in general there is a lot of hesitation because the open model trend is so new.