A survey of data scientists shows barriers to effective working and high levels of job dissatisfaction in some areas. The survey, conducted by analytics software company SAS, also identifies strategies to empower data scientists and organisations.
Pandemic’s impact on data science work
More than 90% of respondents indicated the importance of their work as data scientists was the same or greater compared to before the Covid-19 pandemic. The survey also shows the pandemic upended standard business practices, shifting the assumptions and variables in models and predictive algorithms and causing a ripple effect of adaptations in processes, practices and operating parameters.
Nearly three-quarters (73%) said they are just as productive or more productive since the pandemic, while a similar proportion (77%) said they had the same or greater collaboration with colleagues. This suggests many of the challenges highlighted were in existence, possibly to a greater degree, before the pandemic.
Application of analytics in organisations
More than 66% of respondents were satisfied with the outcomes from analytical projects. However, 42% of data scientists were dissatisfied with their company’s use of analytics and model deployment, suggesting a problem with how analytical insights are used by organisations to inform decision making. This was backed up by 42% saying data science results were not used by business decision makers, making it one of the main barriers faced.
Data science skills gaps
Less than 33% of the respondents reported having advanced or expert proficiency in program-heavy skills, such as cloud management and database administration. This is an issue given that use of cloud services is up significantly, with 94% saying they experienced the same or greater use of cloud since the Covid-19 outbreak.
“Organisations must realise that investing in a team of data scientists with complementary skills could reap huge value for the business, so the cost of hiring needs to consider the return on that investment as we move to significantly more digital and AI-driven business processes,” said Dr Iain Brown, head of data science at SAS UK and Ireland.
Analytical bias and ethics
43% of respondents indicated that their organisation does not conduct specific reviews of its analytical processes with respect to bias and discrimination, and only 26% of respondents reported that unfair bias is used as a measure of model success in their organisation.
Data preparation time
Other challenges experienced were the amount of time spent on data preparation versus model creation. Respondents are spending more of their time (58%) than they would prefer gathering, exploring, managing and cleaning data.
The full report, Accelerating Digital Transformation, explores additional findings and discussion points, including strategies for data scientists to improve processes and outcomes.