QAA: MOOCs and Quality: A Review of Recent Literature
Dr Sarah Hayes July 2005

Definitions

QAA: Quality Assurance Agency for Higher Education. An organisation “to safeguard the standards of awards and improve the quality of UK higher education.”

Notes

Preface

QAA is pro MOOC for innovation, wider participation, life long learning, blended learning opportunities and promoting HE.
QAA “want to help prospective students make informed decisions about the quality of MOOCs, while respecting the open and innovative nature of this provision”
“ UK universities and other awarding organisations are responsible for the quality of all the courses they offer. Since MOOCs are typically non-credit bearing and have no particular entry requirements, they are not formally scrutinised during QAA review.”
QAA wishes to help support and promote the use of MOOCs in HE.
There is a development toolkit that can be used in junction with the “UK Quality Code for Higher education” to assist HEIs and their questions on MOOCs.

Intro

Simpson 2013 sees MOOCs to be designed for increased engagement
Clow 2013 sees MOOCs to “funnel participation”

There are questions on weather MOOCs are a suitable platform to award accreditation in a HE context
“Wintrup et al (2015, p 31) report participant accounts of the value of the ‘unconditional and free nature of their learning’ and ‘intellectual stimulation and personal development’.”
Currently credit is not a key motivator however OBHE 2013 found that this will change.

“MOOCs are significant catalysts for sector wide change” and HEIs need to plan to adapt; considering: organisation, content, delivery, feedback mechanisms, awards, pedagogy, course design, assessment and the teacher’s role.

“MOOCs were successful in: ‘enabling many participants to feel engaged in intellectual endeavours, such as forming new understandings, making connections with previous knowledge and experience, and exploring knowledge actively, creatively and critically’ (HEA, 2015”)

Nature of MOOCs

“In 2014 approximately 1,000 MOOCs were available from universities in the USA, and 800 from European institutions, and in several languages besides English (Bates, 2014).”

There are many different views on MOOCs rise in popularity:
Barber et al, (2013) see MOOCs as revolutionary
Christensen (2010) sees MOOCs as disruptive
Brabon (2014) sees MOOCs as “opening up education on an unprecedented scale”
Bates (2013) sees MOOCs as a “more modern version of educational broadcasting, which does not affect the basic fundamentals of education”

Implication of MOOCs into the process of higher learning:
“Brabon (2014) raises questions about pedagogy, experience, new business models for higher education and quality assurance, suggesting that if MOOCs are simply integrated into institutional flexible and blended learning strategies this overlooks the opportunity to reconceive how higher education might respond, by reimagining the idea of the campus degree.”
“In response, Kernohan (2014, p 7) suggests the majority of commercial MOOCs are closer to traditional models of mass higher education, citing the role of the ‘rockstar professor’ and the focus on the teacher, rather than the institution they work for, where the ‘persistent nature of the teacher as the primary point of contact’ is a design feature of the MOOC and not of quality assurance processes.”

Types of MOOC

Massive Open Online Course, so its infinitely scalable, available freely for all, online or potentially on an intranet and it’s educational material usually structured to fit a learning objective.

cMOOC is connectivist facilitating community learning.

xMOOC is content-orientated, information delivered in the classic classroom format.
Clarke (2013) outlines 8 main types:

Transfer MOOC: an existing course transferred to the MOOC platform

Made MOOC: Innovative creative use of media and the MOOC platform to create a formal, quality-driven MOOC. Peer assesments are used to cope with large nnumbers of prticpants. Format used for Vocational Open Online Courses (VOOCs)

Synchronous MOOC: fixed start and end dates with deadlines for assessments.

Asynchronous MOOC: start and end is flexible or frequently run, learners can complete at their own pace.

Adaptive MOOC: Use of algorithms to modify the course content and format based on student’s prerequisites

Group MOOC: participants are split into small groups depending on location and ability level and they progress through the course together

Connectivist MOOC: content and knowledge sharing within the community of learners.

Mini MOOC: short intense courses hours/days not weeks with a specific learning objective.

Concerns

Quality of learning and the course needs to be evaluated
Delivered in small chunks which are then signed off
Taking an old syllabus and using a MOOC platform will modify the material and delivery
Drop out rates could be due to poor quality or maybe dependant on the students personal needs and objectives
Downes states MOOCs are process defined not outcome defined
Metrics and data analytics can help make a MOOC better and possibly decrease drop out rates
Success of a MOOC is that it meets the learner’s objectives and parts of the course or the assessment element of the course may not be what they need or want; this doesn’t mean they don’t benefit from the course.
Quality, sustainability, pedagogy and completion rates are key elements that concern HEIs using MOOCs.
Working on ensuring the pre-course information on the format and learning objectives of a course may help reduce drop out rates.

Requirements

Conole 2013 says key elements for learning design should promote: reflection, dialogue, collaboration, apply theory to practice, community, creativity, and motivation.
Margaryan et al 2015 conducted a study of MOOCs “based on a 10-principle framework, they draw five fundamental principles from Merrill (2002, 2009, 2013), abstracted from key instructional design theories and models”. First 5 are activity based and the remaining 5 are resources focused:

1. Problem-centred: “acquire skills in the context of real-world problems. This is contrasted with topic-centred instruction.”
2. Activation: “activate existing knowledge as a foundation for new skills.”
3. Demonstration: “showing learners how to apply the new information or skill in new situations. “
4. Application: “apply their newly acquired skill to solve problems.”
5. Integration: “to reflect on what they have learned, revise, synthesise, or modify their new skills, and demonstrate and defend their new knowledge or skill to peers and others.”
6. Collective knowledge: “learners contribute to the collective”.
7. Collaboration
8. Differentiation: each learner is “provided with different avenues of learning, according to their need.”
9. Authentic resources: “resources are drawn from real-world settings.”
10. Feedback: “expert feedback on their performance.”

“’The results indicate that although most MOOCs are well-packaged, their instructional design quality is low’ (Margaryan et al, 2015, p 77).”
“Many MOOCs though have little or no qualified tutoring or guidance, just online areas for student communication and learning materials resulting in learning engagement being out of the control of the organisers.”
Morris 2014 covers 7 main quality issues to consider with MOOCs; but in short everything has issues in maintaining/regulating/monitoring/verifying quality of the MOOC’s content, resources and qualifications.

Oppertunities

MOOCs affecting the educational market gives opportunity to re-evaluate:

• Curriculum, curriculum design and the resources used
• Policy
• Tools
• Relationship with students and other staff
• How MOOCs could fit in with classroom and distance learning

“Wintrup et al (2015) suggest further research from three key perspectives”

• • Education enhancement: curriculum developers and learners
• • Higher education providers and their marketing teams
• • Researchers and policy makers.

See paper for full list of areas in need of further research.
​

​https://docs.google.com/presentation/d/1ozD8RV2Mu_G36IFGpZH631FEWZ6fUOsf6uOTJH_Z1LA/edit?usp=drive_web

​JISC centis: centre for educational technology & interoperability

MOOCs and Open Education: Implications for  Higher Education: A  white  paper 

By  Li  Yuan  and  Stephen  Powell

March 2013

http://publications.cetis.ac.uk/2013/667 (accessed 15/09/2016 17:00)

This summary does not cover this white paper in detail, however I highly recommend reading it regardless that some of the information is out of date regarding revenue streams, figures and the status of some MOOC suppliers. This paper covers how MOOCs impact the Education market and HEIs in particular; covers the current players and how they interact and grow with or separate to HEIs and how HEIs can potentially utilise MOOCs. A clear message that MOOCs are here to stay and the UK educational institutes need to find new business plans, perspectives and possibly pedagogies to adapt to how MOOCs have disrupted the landscape of education.

Definitions

White Paper: (in the UK) a government report giving information or proposals on an issue.

Higher Education: (in terms of UK) non mandatory education after GCSEs including AS/A2 levels, undergraduate and post-graduate

Course Provision: The scope and resources for a course and an outline of the responsibilities and expected outcomes. A specification of the course, a guarantee from the course provider. Regulations and quality proceedures are planned and outlined to ensure the Course provision is met.

Theory of Disruptive Innovation (Bower & Christensen 1995): innovations delivered in a way that goes against the market expectations.

Venture Capitalists: “an investor who either provides capital to startup ventures or supports small companies that wish to expand but do not have access to equities markets.” – investopedia.com

Open Education: broadened access to education.

Disaggregate: to separate into component parts.

Revenue Streams: sources of incoming funds.

Pedagogy: “method and practice of teaching, especially as an academic subject or theoretical concept.” – Google

Processed-based approach: Focusing on the activities and interactions within the course/learning

Content-based approach: Focusing on the resources and layout of the course.

Philanthropic: “a person or organization seeking to promote the welfare of others” – Google. This is often shown by the giving or raising of money for a charitable or humanitarian cause.

Sustainability: the ability to maintain a behaviour or condition indefinitely.

Autonomous Business Unit: A way to coordinate multiple resources into one company or network which internalises competition and analysis; making the whole independent of the current market.

Open Education: Initiatives that broaden the access to educational services/materials.

Copyright: “exclusive and assignable legal right, given to the originator for a fixed number of years, to print, publish, perform, film, or record literary, artistic, or musical material.” – Google.

OCR: a leading UK awarding body, providing qualifications for learners of all ages at school, college, in work or through part-time learning programmes.

OER: Open Education Resources

HEI: Higher Education Institutes

MOOC: Massive Open Online Course

cMOOC: “connectivist MOOCs which are based on a connectivism theory of learning with networks developed informally” “collaborative learning” of like-minded individuals.

xMOOC: “content-based MOOCs which follow a more behaviourist approach” “instructional pedagogy” mirroring the institutions with video lectures and testing.

Sustaining innovations: ones that improve existing services/products, used by HEI.

Disruptive innovations: ones that create a new market, by combining new technology and their business model to target: lowering the price of a service, a new consumer base or an alternative need of the current consumer base, used by start-ups.

Undershot customers: customers willing to pay for improvements of an existing service/product

Overshot customers: where a customers needs are more than provided fro by the existing service/product

UKeU programme: Was an online learning solution that was supply driven not demand led, had low participation and flopped.

Homogeny: a form similar to a common origin, so in this context learning that is similar to the traditional medium and structure, e.g. an online lecture

Diversity: a range of forms, so in this context many ways of learning with different formats and sources for materials. 

TELMap 2012 higher Education scenario Traditional University: homogenous higher education delivered face to face

TELMap 2012 higher Education scenario Unidiversity: diverse higher education delivered face to face

TELMap 2012 higher Education scenario Hybrid University: homogenous higher education delivered online

TELMap 2012 higher Education scenario Online University: diverse higher education delivered online

Open Curriculum: mix resources for autonomous self directed learners

Open Learning: teacher and learners “generate and share new ideas”-in the-“learning process”, aimed at self determined independent learners

Open Assessment: “p2p or crowd sourced assessment with on demand accreditation”

Open Platform: create and maintain UI with open standards to easily exchange information and data

Demographic: The target or current pool of users/consumers of a product or service

Business Strategy: plan considering finances and requirements for a product or service

Business Model: analytical approach to the way a product is developed covering behaviours towards the current market, safe guards and considerations. This can cover company culture and values.

Strategic Plan: step by step plan and any possible alternative routes to achieve a goal

Notes

This paper covered the MOOCs as a disruptive innovation and how the current HEI work with MOOCs an the future potential and requirements of MOOCs as they change the HE market.

Key benefits of MOOCs that fill the needs of HEIs

Globalization of education, experimentation with new business models and pedagogy, lowering cost to the HEI and increasing the affordability to the student, cheap data analysis on how students learn/course material/delivery methods, branding and marketing of the university and it’s courses to improve awareness and reputation, being able to provide credit for students who fall short in their course requirements, extra revenue streams from certification or course access, material easily adapted and deployed in light of course changes, to be able to provide many different formats of material so the learner has choice in learning style, scalable to accommodate huge amounts of students, nurturing partnerships with commercial organizations to increase networks and student recruitment levels, ability to detect under performing or undesirable courses to then either renovate or remove it, supply education to life long learners, enables connections with social media, ability to collaborate with other institutions/educators/learners locally and globally.

Main pedagogies

xMOOC, cMOOC non-profit, cMOOC for profit.

Different business model features

Fees for certificates, fees for participation, fees for access, fees for grading, tuition fee on credited courses, fees to access social media features, selling student information, advertising, course sponsorship, enterprises pay to run their course, applicant screening, human tutoring, human assessment marking, job match services / recruiting tools, appeal for philanthropic donations, non profit.

Concerns raised for HEIs utilising MOOCs

Format suitable only for tech-literate teachers and students, difficult to know what pedagogy is suitable for MOOC format for given course material, difficult to guarantee quality teaching, possibly ambiguous responsibility for the learning process and the examination standards, examination marking resources to cope with potential increase in students and/or assessment submissions, sustainability issues,  motivation of learners, adopting the format poorly or with the wrong strategy could dramatically damage the learning experience and potentially the reputation of the course, budget concerns especially for the initial setup costs, increase of peer engagement/assessment/support could lead to increase of plagiarism or cheating

Elements for my project to consider

This paper outlines the key users as Educators and Ventre Capitalists. In light that the paper is really a strong argument to encourage HEIs to devise a plan and a pedagogy in which to adapt to utilizing MOOCs, and it’s evident that they have not been integrated into course delivery yet, I will focus on HE educators as my target user. I hope to address the concerns that this medium is harder for non tech literate staff and students, hopefully making an east guide to facilitate the creation of an academic MOOC.

It would seem wise to create personas for different potential users, staff, students and non-students if applicable MOOC is not internal use only; use published their demographic studies from MOOCs if available.

Looking into how MOOCs may compliment other teaching processes would be useful to note.

To outline the current differences in style, content and features of those MOOCs in collaboration with universities and those not, should show which features to potentially exclude from the guide.

Copyright requirements, licencing requirements and security guidance should be covered.

Considering I will be focused to provide guidance for developing MOOC as an extension, element or alternative to HEI course material I will be looking at the xMOOC format and those MOOCs currently using it. cMOOCs are a consideration as a compliment to course but is not suitable for early stages of integration into course provision.

Extra

Just want to share a my favourite quote from this paper: “ The development of MOOCs is rooted within the ideals of openness, that knowledge should be shared freely, and the desire to learn should be met without demographic, economic or geological constraints.”

​Autotrader Agile Nights talks 06/09/2016

These are my notes from the talk I went to at AutoTrader’s Agile Nights event.

Talk 2: Organisational Effectiveness: Doug Talbot from Ocado

Before my notes begin I would like to say that this is more a talk about his very interesting role as a Catalyst.

Doug Talbot is a Catalyst at Occado Technology; Occado makes their own distribution software, technology and robotics; they also develop and retail a selling platform that help similar companies with their logistics and robotics. 

A Catalysts are a team that look at psychology, coaching, Lean + Agile methodologies, tech and maths to improve the effectiveness of the workplace and the company. They found that Trust is the most important thing in the squads, also the needs of the people, collaboration and the ability to adapt and reallocate resources are important. Getting the right people, the right thing being built right is a formula for great business.

Going into an environment and teaching the process of a methodology is not enough, you need leadership to support, improve, maintain and demonstrate the culture and its ideals. 

Being a Catalyst is to give the environment and tools for the people to develop their own practices.

Evidence Based Management Model: Ask, Search, criticise, implement, evaluate. Repeat. Using this model the Catalysts try and test little changes in individual squads to see the effects to see if it should be introduced to the rest of the squads. 

Psychology: Complexity theory, find solid science to help, theory or constraints, Real Options Maths Scholes theory, Behavioural Economics, Complex adaptive systems. They use psychology to research new and effective ways to help the culture, and back these ideas with facts.

Ocado Catalyst Initiatives: Tribe/Product organisation, measuring outcomes, values, scrapping annual appraisals for more continuous communication, leadership training, developing a promotions model, situational interviewing techniques, peer feedback, agile coaching, having leaders focused on skills over efficiency.

J. Richard Hackman wrote Collaborative Intelligence; this book promotes peer coaching to tackle dynamic and complex problems. Trust is important in your team, far more important than adhering to practices. 

80% of developers think they are above average; but the truth is that there is no standard, no objectivity that you can measure against, so it’s impossible to evaluate. However if you focus on teamwork, that you can measure with peer review and questionnaires which you can measure you’ll get quality work. EQ is more important and effective than IQ. 

Many books give lots of advice and sometimes this advice is contradicting, this is because you need to try different things out and find what fits your team and your environment. Developing software is not a one size fits all, it has different ways of doing everything varying in logic, techniques and solution and so are the teams. Spotify uses Squads, Gore uses Leadership, Valve uses extreme job-crafting.

When researching psychology make sure you are looking at what’s relative; looking at research done with mobile app developers may not be relative to those maintaining a legacy product. Catalyst teams are only in charge of optimising the people not the practice. However bear in mind that there is no perfect solution, it’s a delicate changing balance to continually improve.

​Autotrader Agile Nights talks 06/09/2016

These are my notes from the talk I went to at AutoTrader’s Agile Nights event.

Talk 1: Mind the Gap: Nikki Barton from Autotrader

Visual Aid: 1925 full length film Battleship Potempkin – Sergio Einsenstien. This film was  the pioneer of montage and the main scene for this is “The Odessa Steps” a 7 minute montage that Sergio focused on the shots and the gaps to create an impactful experience.

However in UX we need to be mindful of cuts and gaps, as it disrupts the user’s experience and this isn’t beneficial. These gaps can occur from switching between platforms (mobile/PC/human interaction); the user needs a seamless experience to reduce distractions. 

Visual Aid: Martin  Vargic’s “Map of the Internet 1.0”. This image is a great expression of how our internet experiences a broken and have gaps, distractions and differences.

The way we access to areas of the internet has changed, there are many journeys and doors to get to the same page; it is no longer typing http://www.____.com.

Development of branding. Branding is not just a logo, it is also layout, access and the way the user uses the product/site/experience. Strong brands are experience-led, brand-experience, giving the user a connection to the brand. E.g the way you interact with sites like Amazon, Facebook, Google.

Ethnographic research done in Autotrader showed people prefer to do research on mobile but when it comes to action they still prefer to do it on a PC. Nikki Barton has seen these results with other products too. Based on this research they designed the web to have long scrolling screens that progressively load, this suited giving a smooth unbroken experience for fulfilling their actions. In contrast the mobile had minimal scrolling, and had a tabbed layout to allow to flick back and forth to compare and view lots of information with ease; also some features were removed to reduce clutter if it was unlikely that it was needed.

Autotrader also used google analytics to look at the drop off rates along the journey; this then means you know what stops the customer from going forward, you can see if there’s any gaps in the experience, a disruption or blocker to a user will cause these drop offs.

Designer’s can also intentionally create gaps; when 2 companies merge the experiences can clash or no align, this can create a confusing unpleasant experience for the user. The solution is often to keep both separate until the technology or design can adapt to accommodate the best of both, e.g. Nokia and Windows. Keeping the experiences separate can also allow for A B testing and QT as the two products merge.

Design should be Content and Experience focused rather than on checklists of SCRUM, Agile, Waterfall, etc. Being held to these checklists can hinder exploration which in turn will stifle creative thinking, being experience led, curiosity, to be successful you need the squad/workspace to be a haven for new ideas to be pitched.

The desirability of the product is important. Product should be human, HCI-focused, adapted to how people work, built with empathy for the  user, adapted for use in the intended context for the intended user. It’s necessary to observe your users and to use your team’s designers, as designers are experts in observation. 

We should design for the future, for future tech and future requirements, e.g. voice UI, VR.

“Culture eats strategy for breakfast” there is a challenge to keep the product united across many squads but with a healthy open culture in the workplace this is easily achieved with similar ideals and trust in place.

I read this book and here are my notes.

Definitions

Elearning: “is when we use computers and the networks to which these are linked to in someway support the learning process” (This is very vague however this is explained as the nature of eLearning, it is vague and continually changing in its scope)

CBT: Computer Based Training (This was the earliest form of eLearning)

Synchronous Communication: real-time learning where all learners access the material at the same time e.g. webinars

Asynchronous Communication: not real-time where the material can be accessed on demand e.g. forums 

Blended Learning: “using forums for group discussions, blogs, as a means of reflective learning and wikis for group tasks”.  – Chapter 1

“The combination of different training media” “to create an optimum trainign program for a specific audience. The term “blended” means the traditional instructor-led ttrainign is suupplemented with other electronic formats” – Josh Bernsin 2004

Blending learning is “a complementary mix of teaching methods delivered using an appropriate selection of related resources, often technology based resources”.  Chalk + Talk.

Exposition: “is the delivery from teacher or subject expert to learner” this is a “one way process” which is “top down and teacher centred”. It is that the “person who is delivering that determines what is delivered and how”; either by the context of an event like a lecture, or in the form of content like a book.

Instruction / process of Instructional design: this is objective focused rather than learner goal focused; it is adaptive to the learners ability e.g. on job training or workbooks

Guided Discovery: is “an inductive process, it leads the learner towards insights and generalizations rather than providing these on a plate”, it “rarely has specific learning objectives”. E.g. “experiments, simulations, case studies, team building exercises”. The learners are “presented with a task to accomplish”, after the task there is reflection and analysis on the task and the results.

Exploration: “learner determines their own learning process, taking advantage of resources” this can be prompted by a starting point or trigger e.g. a handout or list of recommended books.

Informal Vs Formal Learning: “formal learning is like riding a bus: the driver decides where the bus is going; the passengers are along for the ride. Informal learning is like riding a bike: the rider chooses the destination, the speed, and the route.” E.g. Formal : planned classes, Informal natural on the job. Informal learning can be intentionally, accidental or unconsciously. 

eLearning Notes

Learners like their own pace, small chunks and learning on demand.

“Participants are more likely to carry out a self paced task such as reading or writing an assignment, if they know a live event is coming up that they will have to report their progress” (this is backed up by the TED Talk “Cultivating an E-Learning Culture | Lloyd Espiritu | TEDxDLSU”)

Digital Learning content asset Types: Interactive tutorials/modules, simulations, video / audio, online presentations, job aids / reference documents, Games/  data-driven modelling, collaborative activities, mobile apps.

Modern blended learning: has multiple routes, practice required, informal format, allows revisiting of material at any time in any location, displays the same content in many different ways, can evolve and adapt, requires application, is embedded into the learners workflow, allows for contribution to its curriculum and resources, one can share with others, includes reflection, and is learner controlled / learner led.

Tip: Don’t blend too many types and don’t put all material formats on one platform (e.g. a VLE) instead use a Hub for all the resources (e.g. Facebook).
Project Management Notes:

Project Management Notes

Popular project management methodologies such as Waterfall, PRINCE2, Agile, eXtreme and ADDIE which are incremental/phased/iterative ways to handle an eLearning project.  ADDIE was covered in depth which stands for: Analyse, Design, Develop, Implement, Evaluate; although this was the recommended method I believe I will stick to my preferred which is  Agile, as it is the industry standard.

The following questions were recommended to be asked before starting the project: What? Why? Constraints? Project Breakdown? Who’s doing What? What tools/systems? Communication format? What happens after?

“Summary of Clive Shepherd’s Blended Learning Cookbook”. Step 1: “Analyse the learning requirement”. Step 2 : “Design Your Learning Strategy”, this is via GEARR + exercise and practice. GEARR is an adaption of the Microsoft MODL, stands for Gather, Expand, Apply, Reflect, Review. Gear: background and core information on learner. Expand: add to your gathered information. Exercise and Practice: run exercises and simulations to gain context and motives for your target learners. Apply: find the “moment of need” when is the content of the course used or needed in context. Reflect: gather feedback. Review: review the impact, effectiveness, contributions, performance, and actively encourage your participants to review these elements. Step 3: “Selecting a mix off learning resources”. There are 3 types of activities: ABSORB: when they interact with new information. DO: when they use the information to achieve a learning outcome. CONNECT: when they make connections between info using information + skill + situations + abilities to then facilitate reflection, collaboration and sharing.

Tip: Use the Blended Learning Design Model to Analyse a current teaching strategy and adapt from there.

Disclaimer

I skipped chapters 2,3,5,some of chapter 7, 8 and 9; this was because these were focused on how to integrate eLearning into the workplace which was not relevant to my unit or my project. Chapters 10 and 11 were also skipped as they covered Game-Based Learning and Metrics which I will revisit if suitable.

Found out that as part of my course I will be keeping a blog as part of my E-Learning unit. Next academic year I’ll be studying  E-Learning, HCI, Data Engineering and working on a dissertation about MOOC design. I’m very excited, so I thought to dust off my old blog and spruce it up.