It seems that you'd be able to do this reasonably simply by using polar coordinates. Find, in polar coordinates, the position of all 3 of your points A, B and C. Since they lie on the surface of a sphere, their "length" component will always equal to the radius of the sphere. You need to then calculate the two angles for each point.
If you don't know how polar coordinates work, it's identical to the concept of declination and right ascension used in astronomy. To represent a point in 3D space, you can use 2 angles and a length. If you think of the night sky, it's like a big sphere surrounding the Earth. To locate a point on the night sky, you use one angle to represent the angle from North, and you use one more angle to determine how far up from the horizon that point is. You can apply the same principles and some simple trigonometry to determine polar coordinates from a set of cartesian coordinates.
Once you have all 4 of your points in polar coordinates, you can just compare their angles to ensure that they're all within each other. For example, let's demonstrate with a simpler problem: a regular 2D circle of radius 10. I have 2 points on the edge of my circle, A and B. A is located 25 degrees around the circle, B is located 50 degrees around the circle.
I want to know if point X, at a distance of 5 from the centre of the circle and an angle of 30 degrees, is within the arc bounded by points A and B. Since the distance is 5 (and the radius of the circle is 10), we know the point is inside the circle somewhere. And since the angle of point X (30 degrees) is between the two angles bounded by A and B (25 and 50 degrees, respectively), it's within the arc bounded by points A and B. Thus, point X is within the specified area.
You can generalise this same method to work in 3D, it's just that you'll have 2 angles to compare per point, rather than just 1. It would certainly be possible to do this in cartesian, but it seems like it would be easier using polar.